In recent times, the field of agriculture has been in urgent need of modernizing, since the amount of manual work people need to put in to check if plants are growing correctly is still highly extensive. Despite several advances in agricultural technology, people working in the agricultural industry still need to have the ability to sort and recognize different plants and weeds, which takes a lot of time and effort in the long term.
The potential is ripe for this trillion-dollar industry to be greatly impacted by technological innovations that cut down on the requirement for manual labor, and this is where Artificial Intelligence can benefit the workers in this field, as the time and energy required to identify plant seedlings will be greatly shortened by the use of AI and Deep Learning. The ability to do so far more efficiently and even more effectively than experienced manual labor could lead to better crop yields, the freeing up of human involvement for higher-order agricultural decision making, and in the long term will result in more sustainable environmental practices in agriculture as well.
AI Generated Image
To classify the plant seedlings into their respective categories.
The Aarhus University Signal Processing group, in collaboration with the University of Southern Denmark, has provided the data containing images of unique plants belonging to 12 different species.
There are 12 plant species (categories) in the dataset:
#connect to google drive
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
#parent directory where images are stored
image_file = '/content/drive/MyDrive/Projects/ML/Data/plantSeedlings_images.npy' #npy file containing images
label_file = '/content/drive/MyDrive/Projects/ML/Data/plantSeedlings_Labels.csv' #csv file containing the labels for each image of the npy file
#visualkeras is a library to visualize cnn architectures
!pip install visualkeras
Requirement already satisfied: visualkeras in /usr/local/lib/python3.10/dist-packages (0.0.2) Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from visualkeras) (9.4.0) Requirement already satisfied: numpy>=1.18.1 in /usr/local/lib/python3.10/dist-packages (from visualkeras) (1.23.5) Requirement already satisfied: aggdraw>=1.3.11 in /usr/local/lib/python3.10/dist-packages (from visualkeras) (1.3.16)
#library for creating data paths
import os
#library for randomly selecting data points
import random
#library for performing numerical computations
import numpy as np
#library for data manipulation and analysis
import pandas as pd
#library for data visualization
import seaborn as sns
#library for creating and showing data
import matplotlib.pyplot as plt
#library for reading and showing images
import matplotlib.image as mpimg
#Python Imaging Library to handle image files
from PIL import Image
#OpenCv for image and video processing
import cv2
#sklearn
from sklearn.model_selection import train_test_split
from sklearn.utils import class_weight
from sklearn.utils.class_weight import compute_sample_weight
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
from sklearn.preprocessing import LabelBinarizer
#tensorflow
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.preprocessing.image import img_to_array, load_img
#importing all the required sub-modules from Keras
from keras import backend
from keras.utils import to_categorical
from keras.models import Sequential, Model
from keras.applications.vgg16 import VGG16
from keras.preprocessing.image import ImageDataGenerator
from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, BatchNormalization, Dropout
from keras import losses, optimizers
from keras.optimizers import RMSprop,Adam,SGD #optimiers for optimizing the model
from keras.callbacks import EarlyStopping, ModelCheckpoint #regularization method to prevent the overfitting
from keras.utils.vis_utils import plot_model
#library to visualize model architecture
import visualkeras
#ignore warnings that may occur due to deprecations
import warnings
warnings.filterwarnings("ignore")
First we will load the image_files and their corresponding labels.
#load the numpy array
img_array = np.load(image_file)
#visualize contents of array, index 0
img_array[0]
array([[[ 35, 52, 78],
[ 36, 49, 76],
[ 31, 45, 69],
...,
[ 78, 95, 114],
[ 76, 93, 110],
[ 80, 95, 109]],
[[ 33, 46, 68],
[ 37, 50, 73],
[ 48, 65, 83],
...,
[ 81, 96, 113],
[ 74, 89, 105],
[ 83, 95, 109]],
[[ 34, 50, 68],
[ 35, 52, 72],
[ 70, 85, 101],
...,
[ 83, 97, 112],
[ 79, 94, 108],
[ 79, 94, 107]],
...,
[[ 35, 50, 69],
[ 42, 57, 73],
[ 42, 57, 72],
...,
[ 60, 76, 92],
[ 67, 81, 97],
[ 64, 77, 95]],
[[ 36, 52, 67],
[ 48, 63, 78],
[ 41, 57, 73],
...,
[ 44, 66, 83],
[ 58, 76, 91],
[ 57, 74, 90]],
[[ 44, 58, 70],
[ 43, 57, 73],
[ 40, 55, 72],
...,
[ 41, 70, 92],
[ 55, 78, 97],
[ 61, 79, 96]]], dtype=uint8)
#shape of the image array
img_array.shape
(4750, 128, 128, 3)
The image array contains 4,750 records. Each record has a length of 128 rows and 128 columns (resolution in pixels). And each element has three values representing Red, Green, Blue (RGB).
#read the labels and store in a dataframe
label_df = pd.read_csv(label_file)
#display the head of the labels dataset
label_df.head(10)
| Label | |
|---|---|
| 0 | Small-flowered Cranesbill |
| 1 | Small-flowered Cranesbill |
| 2 | Small-flowered Cranesbill |
| 3 | Small-flowered Cranesbill |
| 4 | Small-flowered Cranesbill |
| 5 | Small-flowered Cranesbill |
| 6 | Small-flowered Cranesbill |
| 7 | Small-flowered Cranesbill |
| 8 | Small-flowered Cranesbill |
| 9 | Small-flowered Cranesbill |
#print shape of label dataframe
label_df.shape
(4750, 1)
There are 4750 records in total.
#display information of the label dataframe and its data type
label_df['Label'].info
<bound method Series.info of 0 Small-flowered Cranesbill
1 Small-flowered Cranesbill
2 Small-flowered Cranesbill
3 Small-flowered Cranesbill
4 Small-flowered Cranesbill
...
4745 Loose Silky-bent
4746 Loose Silky-bent
4747 Loose Silky-bent
4748 Loose Silky-bent
4749 Loose Silky-bent
Name: Label, Length: 4750, dtype: object>
We should convert 'Label' to a string type because later we will convert it to one-hot vectors. To convert to one-hot vectors, we will use 'to_categorical' which expects an array of integers or numeric values. Otherwise type may throw an error.
#convert to string type
label_df['Label'] = label_df['Label'].astype(str)
#Number of unique classes (labels)
num_classes = label_df.nunique()[0]
print("The number of seed classes is ",num_classes)
The number of seed classes is 12
#create an array containing the labels of classes
name_classes = label_df['Label'].unique()
name_classes
array(['Small-flowered Cranesbill', 'Fat Hen', 'Shepherds Purse',
'Common wheat', 'Common Chickweed', 'Charlock', 'Cleavers',
'Scentless Mayweed', 'Sugar beet', 'Maize', 'Black-grass',
'Loose Silky-bent'], dtype=object)
#visualize the seedlings label
figsize=(14, 8)
count_plot = sns.countplot(data=label_df, x="Label")
#rotate x-axis labels
count_plot.set_xticklabels(count_plot.get_xticklabels(), rotation=90)
count_plot.set(xlabel='seed')
plt.show()
#Number of items per categories
label_df.value_counts()
Label Loose Silky-bent 654 Common Chickweed 611 Scentless Mayweed 516 Small-flowered Cranesbill 496 Fat Hen 475 Charlock 390 Sugar beet 385 Cleavers 287 Black-grass 263 Shepherds Purse 231 Common wheat 221 Maize 221 dtype: int64
The data is imbalanced. Class Maize and Common Wheat are roughly 1/3 the size of Loose Silky-bent.
#convert dataframe to array
label_array = label_df.to_numpy().squeeze()
#Displaying images randomly
#Generate random indices to select rows x columns images
rows = 4
cols = 5
ix_img = np.random.choice(len(label_array), size=rows*cols, replace=False)
# Select the images and labels
images = img_array[ix_img]
labels = label_array[ix_img].tolist()
# Define the size of the figure
fig = plt.figure(figsize=(12, 12))
# Create a rows x cols grid of subplots
for i in range(rows*cols):
ax = fig.add_subplot(rows, cols, i+1)
ax.imshow(images[i])
ax.set_xlabel(str(labels[i]), fontsize=10)
ax.set_xticks([])
ax.set_yticks([])
# Show the figure
plt.tight_layout()
plt.subplots_adjust(wspace=0.2)
plt.show()
#Displaying images (randomly) from each category
num_images = 5
# Create a dictionary to store the indices of each category
category_indices = {}
for category in range(num_classes):
category_indices[category] = label_df[label_df['Label'] == category].index.tolist()
# Loop through each unique label
for label in name_classes:
# Get the indices of all images with the current label
indices = np.where(label_df['Label'] == label)[0]
# Choose num_images random indices
random_indices = np.random.choice(indices, size=num_images, replace=False)
# Get the corresponding images from "img_array"
images = img_array[random_indices]
# Create a plot of the images
fig, ax = plt.subplots(1, num_images, figsize=(20, 4))
for i, image in enumerate(images):
ax[i].imshow(image)
ax[i].axis('off')
# Set the suptitle and adjust the spacing
fig.suptitle(label, fontsize=16)
fig.subplots_adjust(top=1.1, bottom=0.1)
plt.show()
Output hidden; open in https://colab.research.google.com to view.
From the dataset image analysis we foresee the need to remove noise from the images. To do this we can use Gaussian Blurring.
We will perform Gaussian Blurring to smooth the images by averaging the intensity of each pixel with the intensity of neighboring pixels.
# Choose a random index for the image
index = np.random.randint(0, img_array.shape[0])
# Extract the image and its label
image = img_array[index, :, :, :]
label = label_df.iloc[index, 0]
fig = plt.figure(figsize=(18, 18))
# Plot the original image
plt.subplot(1, 2, 1)
plt.imshow(image.squeeze(), cmap='gray')
plt.title('Original Image (Label: {})'.format(label))
plt.axis('off')
(-0.5, 127.5, 127.5, -0.5)
# Define the kernel size for the Gaussian blur
kernel_size = (3, 3)
# Apply Gaussian blur to each image in the img_array
for i in range(img_array.shape[0]):
img = img_array[i,:,:,:]
img = cv2.GaussianBlur(img, kernel_size, 0)
img_array[i,:,:,:] = img
After Gaussian Blurring we will scale the pixel values to a fixed range (i.e. 0 to 1).
# Convert the data type of the img_array to float32
img_array = img_array.astype('float32')
# Normalize the pixel values to the range of 0 to 1
img_array = img_array / 255.0
# Plot the preprocessed image
fig = plt.figure(figsize=(18, 18))
plt.subplot(1, 2, 2)
plt.imshow(image.squeeze(), cmap='gray')
plt.title('Preprocessed Image (Label: {})'.format(label))
plt.axis('off')
# Show the plots
plt.show()
#Displaying images (randomly) from each category
num_images = 5
# Create a dictionary to store the indices of each category
category_indices = {}
for category in range(num_classes):
category_indices[category] = label_df[label_df['Label'] == category].index.tolist()
# Loop through each unique label
for label in name_classes:
# Get the indices of all images with the current label
indices = np.where(label_df['Label'] == label)[0]
# Choose num_images random indices
random_indices = np.random.choice(indices, size=num_images, replace=False)
# Get the corresponding images from "img_array"
images = img_array[random_indices]
# Create a plot of the images
fig, ax = plt.subplots(1, num_images, figsize=(20, 4))
for i, image in enumerate(images):
ax[i].imshow(image)
ax[i].axis('off')
# Set the suptitle and adjust the spacing
fig.suptitle(label, fontsize=16)
fig.subplots_adjust(top=1.1, bottom=0.1)
plt.show()
Output hidden; open in https://colab.research.google.com to view.
Beware that the differences in the appearance of the original image and the Gaussian-Normalized image are hard to notice by a human eye.
# Define a dictionary to map each label to a unique integer value
label_map = {'Small-flowered Cranesbill':0, 'Fat Hen':1, 'Shepherds Purse':2,
'Common wheat':3, 'Common Chickweed':4, 'Charlock':5, 'Cleavers':6,
'Scentless Mayweed':7, 'Sugar beet':8, 'Maize':9, 'Black-grass':10,
'Loose Silky-bent':11}
# Convert the label strings to integer values using the label map
label_df['Label'] = label_df['Label'].map(label_map)
#splitting data into training and testing set
X_train, X_test, y_train, y_test = train_test_split(img_array,label_df['Label'], test_size=0.1, random_state=42, stratify=label_df['Label'])
We can oversample the minority class by adding more samples to that minority classes doing augmentation, generating synthetic data, etc. We can undersample the majority class by removing samples from the majority class to balance the distribution. However we will lose information and although the distribution of the majority class to minitory class is 3 to 1, this may not be a good approach because our dataset is not very large. We can also use data augmentation. And we can also increase the weights of the minority classes.
The data is imbalanced as we have seen earlier. To avoid bias toward the majority class, or underfitting the minority class, we should balance the data.
Balancing the data on the Training set by adjusting weights.
The main reason to use this technique is because reprocessing the images have a very high computational cost.
# Calculate class weights to balance the data
class_weights = {}
for i in range(12):
class_weights[i] = np.sum(y_train == i) / len(y_train)
print(class_weights)
{0: 0.10432748538011696, 1: 0.09988304093567252, 2: 0.048654970760233916, 3: 0.04654970760233918, 4: 0.1286549707602339, 5: 0.08210526315789474, 6: 0.060350877192982454, 7: 0.10853801169590643, 8: 0.08116959064327485, 9: 0.04654970760233918, 10: 0.05543859649122807, 11: 0.13777777777777778}
# Use the class weights to balance the data
sample_weights = compute_sample_weight(class_weight=class_weights, y=y_train)
y_train.info
<bound method Series.info of 437 0
1571 4
838 1
4040 10
4707 11
..
746 1
1217 3
3058 7
4682 11
787 1
Name: Label, Length: 4275, dtype: int64>
# Convert the labels to one-hot vector (we only encode the y_train and y_test because these are categorical)
y_train_onehot = to_categorical(y_train, num_classes=12)
y_test_onehot = to_categorical(y_test, num_classes=12)
print("X_train shape:",X_train.shape)
print("X_test shape:",X_test.shape)
print("y_train shape:",y_train.shape)
print("y_test shape:",y_test.shape)
print("y_train one-hot encoded shape:",y_train_onehot.shape)
print("y_test one-hot encoded shape:",y_test_onehot.shape)
X_train shape: (4275, 128, 128, 3) X_test shape: (475, 128, 128, 3) y_train shape: (4275,) y_test shape: (475,) y_train one-hot encoded shape: (4275, 12) y_test one-hot encoded shape: (475, 12)
The data is compatible with Keras. img_array is a 4D array with shape (4750, 128, 128, 3), with 4750 images, sized 128x128px and the 3 RGB channels.
We already one hot encoded y_train and y_test, so these are also compatible with Keras.
input_shape = (128,128,3)
#create dataframe to store scores
Scoresdf = pd.DataFrame(columns =['model', 'accuracy', 'layers', 'total parameters', 'neurons', 'optimizer'])
We will first start with a simple model of only 1 Convolution layer.
Model Definition
Layers : 4
Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu
Layer 2: MaxPooling2D layer
Layer 3: Flatten layer
Layer 4: Dense layer : 12 neurons, Relu
Optimizer: Adam
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
# Define the model architecture
model0 = Sequential()
#Convolution with 32 filters size 3,3 and relu
model0.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))
#add max pooling to reduce the size of output of first convolution layer
model0.add(MaxPooling2D(pool_size=(2, 2)))
#reshape the output to a flat vector
model0.add(Flatten())
#add output layer
model0.add(Dense(num_classes, activation='softmax'))
# Compile the model
model0.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model0.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
max_pooling2d (MaxPooling2D (None, 63, 63, 32) 0
)
flatten (Flatten) (None, 127008) 0
dense (Dense) (None, 12) 1524108
=================================================================
Total params: 1,525,004
Trainable params: 1,525,004
Non-trainable params: 0
_________________________________________________________________
Model Visualization
#visualize the CNN architecture
plot_model(model0, to_file='model0_plot.png', show_shapes=True, show_layer_names=True)
#visualize the CNN architecture using visualkeras
visualkeras.layered_view(model0, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 30 epochs and validation_split as 10%
history0 = model0.fit(X_train, y_train_onehot, epochs=30, batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/30 120/121 [============================>.] - ETA: 0s - loss: 1.7592 - accuracy: 0.4187 Epoch 1: val_accuracy improved from -inf to 0.49533, saving model to best_model.h5 121/121 [==============================] - 5s 22ms/step - loss: 1.7579 - accuracy: 0.4195 - val_loss: 1.4393 - val_accuracy: 0.4953 Epoch 2/30 117/121 [============================>.] - ETA: 0s - loss: 1.1295 - accuracy: 0.6242 Epoch 2: val_accuracy improved from 0.49533 to 0.52804, saving model to best_model.h5 121/121 [==============================] - 2s 18ms/step - loss: 1.1295 - accuracy: 0.6249 - val_loss: 1.3444 - val_accuracy: 0.5280 Epoch 3/30 118/121 [============================>.] - ETA: 0s - loss: 0.8919 - accuracy: 0.7076 Epoch 3: val_accuracy improved from 0.52804 to 0.56075, saving model to best_model.h5 121/121 [==============================] - 2s 13ms/step - loss: 0.8918 - accuracy: 0.7065 - val_loss: 1.3244 - val_accuracy: 0.5607 Epoch 4/30 117/121 [============================>.] - ETA: 0s - loss: 0.7481 - accuracy: 0.7591 Epoch 4: val_accuracy improved from 0.56075 to 0.61449, saving model to best_model.h5 121/121 [==============================] - 2s 13ms/step - loss: 0.7480 - accuracy: 0.7593 - val_loss: 1.1457 - val_accuracy: 0.6145 Epoch 5/30 119/121 [============================>.] - ETA: 0s - loss: 0.6163 - accuracy: 0.8120 Epoch 5: val_accuracy improved from 0.61449 to 0.65421, saving model to best_model.h5 121/121 [==============================] - 2s 14ms/step - loss: 0.6152 - accuracy: 0.8123 - val_loss: 1.1489 - val_accuracy: 0.6542 Epoch 6/30 118/121 [============================>.] - ETA: 0s - loss: 0.4893 - accuracy: 0.8620 Epoch 6: val_accuracy improved from 0.65421 to 0.65654, saving model to best_model.h5 121/121 [==============================] - 1s 12ms/step - loss: 0.4904 - accuracy: 0.8617 - val_loss: 1.1100 - val_accuracy: 0.6565 Epoch 7/30 117/121 [============================>.] - ETA: 0s - loss: 0.3744 - accuracy: 0.8993 Epoch 7: val_accuracy did not improve from 0.65654 121/121 [==============================] - 1s 11ms/step - loss: 0.3717 - accuracy: 0.8999 - val_loss: 1.2097 - val_accuracy: 0.6262 Epoch 8/30 117/121 [============================>.] - ETA: 0s - loss: 0.3120 - accuracy: 0.9241 Epoch 8: val_accuracy improved from 0.65654 to 0.66121, saving model to best_model.h5 121/121 [==============================] - 1s 12ms/step - loss: 0.3107 - accuracy: 0.9244 - val_loss: 1.1094 - val_accuracy: 0.6612 Epoch 9/30 119/121 [============================>.] - ETA: 0s - loss: 0.2301 - accuracy: 0.9485 Epoch 9: val_accuracy did not improve from 0.66121 121/121 [==============================] - 1s 11ms/step - loss: 0.2295 - accuracy: 0.9488 - val_loss: 1.2245 - val_accuracy: 0.6589 Epoch 10/30 117/121 [============================>.] - ETA: 0s - loss: 0.1703 - accuracy: 0.9682 Epoch 10: val_accuracy did not improve from 0.66121 121/121 [==============================] - 1s 12ms/step - loss: 0.1722 - accuracy: 0.9670 - val_loss: 1.2595 - val_accuracy: 0.6075 Epoch 11/30 118/121 [============================>.] - ETA: 0s - loss: 0.1336 - accuracy: 0.9783 Epoch 11: val_accuracy did not improve from 0.66121 121/121 [==============================] - 1s 11ms/step - loss: 0.1342 - accuracy: 0.9782 - val_loss: 1.2517 - val_accuracy: 0.6425 Epoch 12/30 117/121 [============================>.] - ETA: 0s - loss: 0.0929 - accuracy: 0.9904 Epoch 12: val_accuracy did not improve from 0.66121 121/121 [==============================] - 1s 12ms/step - loss: 0.0926 - accuracy: 0.9906 - val_loss: 1.3501 - val_accuracy: 0.6145 Epoch 13/30 119/121 [============================>.] - ETA: 0s - loss: 0.0705 - accuracy: 0.9937 Epoch 13: val_accuracy did not improve from 0.66121 121/121 [==============================] - 1s 12ms/step - loss: 0.0708 - accuracy: 0.9938 - val_loss: 1.2872 - val_accuracy: 0.6495 Epoch 14/30 116/121 [===========================>..] - ETA: 0s - loss: 0.0489 - accuracy: 0.9976 Epoch 14: val_accuracy did not improve from 0.66121 121/121 [==============================] - 2s 13ms/step - loss: 0.0483 - accuracy: 0.9977 - val_loss: 1.3812 - val_accuracy: 0.6379 Epoch 15/30 116/121 [===========================>..] - ETA: 0s - loss: 0.0337 - accuracy: 0.9992 Epoch 15: val_accuracy did not improve from 0.66121 121/121 [==============================] - 2s 13ms/step - loss: 0.0335 - accuracy: 0.9992 - val_loss: 1.3267 - val_accuracy: 0.6449 Epoch 16/30 118/121 [============================>.] - ETA: 0s - loss: 0.0285 - accuracy: 0.9981 Epoch 16: val_accuracy did not improve from 0.66121 121/121 [==============================] - 1s 12ms/step - loss: 0.0285 - accuracy: 0.9982 - val_loss: 1.3954 - val_accuracy: 0.6519 Epoch 17/30 120/121 [============================>.] - ETA: 0s - loss: 0.0229 - accuracy: 1.0000 Epoch 17: val_accuracy did not improve from 0.66121 121/121 [==============================] - 1s 12ms/step - loss: 0.0229 - accuracy: 1.0000 - val_loss: 1.4152 - val_accuracy: 0.6449 Epoch 18/30 121/121 [==============================] - ETA: 0s - loss: 0.0156 - accuracy: 1.0000 Epoch 18: val_accuracy did not improve from 0.66121 121/121 [==============================] - 1s 12ms/step - loss: 0.0156 - accuracy: 1.0000 - val_loss: 1.4935 - val_accuracy: 0.6379 Epoch 18: early stopping
Notice that at Epoch 16 the training stopped since the validation loss had continued to increase during 10 epochs. The highest validation accuracy was 0.65654 and validation loss 1.2791.
Model0 - Accuracy vs Epoch curve
plt.plot(history0.history['accuracy'])
plt.plot(history0.history['val_accuracy'])
plt.title('model0 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
We can see that this model performs poorly. The model performs very well on training but very poorly in the validation. It is clearly overfitting.
Evaluation of model0 model.evaluate using the Test data
# Evaluate the model on the test data
score0 = model0.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score0[0])
print('Test accuracy:', score0[1])
Test loss: 1.6907126903533936 Test accuracy: 0.648421049118042
# Test Prediction
y_pred = model0.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 6ms/step
# Test Accuracy
accuracyScore0 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore0)
0.6484210526315789
Classification Report
# Compute the classification report
cr0 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr0)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.76 0.84 0.80 50
Fat Hen 0.61 0.62 0.62 48
Shepherds Purse 0.42 0.22 0.29 23
Common wheat 0.44 0.32 0.37 22
Common Chickweed 0.93 0.67 0.78 61
Charlock 0.83 0.97 0.89 39
Cleavers 0.68 0.52 0.59 29
Scentless Mayweed 0.52 0.81 0.63 52
Sugar beet 0.64 0.47 0.55 38
Maize 0.59 0.59 0.59 22
Black-grass 0.36 0.38 0.37 26
Loose Silky-bent 0.65 0.72 0.69 65
accuracy 0.65 475
macro avg 0.62 0.60 0.60 475
weighted avg 0.66 0.65 0.64 475
Charlock has a higher precision, recall, and F1 score (0.88, 90, 0.89) than all other classes. Black-grass has the opposite (all low: 0.36, 0.46, 0.41).
Model0 has an overall accuracy of 0.65, it correctly predicted the seedling's class for 65% of the images in the test set.
The precision, recall, and F1-score for each class vary, with some classes having good performance (e.g., Charlock) and others having poor performance (e.g., Shepherds Purse).
The macro average F1 score is 0.65, which means that Model0 is not so good overall. The weighted average F1 score is also .65 and this means that the model is not biased towards any particular class.
Scores_Models_df = Scoresdf.append({'model':'Model0','accuracy':accuracyScore0,'layers':'4','total parameters':'1,525,004', 'neurons':'12', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Set the confusion matrix
cm0 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm0, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Model0 is able to predict some classes with good accuracy but others are not predicted well. We can build better models to improve the performance on the other classes.
This model trained 1,525,004 parameters.
We called our first model, Model0 because this was a simple model with only one Convolution layer, so we didn't expect much from it.
Model Definition
This model builds on Model0. Model1 adds a second Convolutional layer and the fully connected layer has 128 neurons to improve the previous Model (Model0).
Layer 7: Droput layer: 0.5
Optimizer: Adam
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
# Define the model architecture
model1 = Sequential()
#Convolution with 32 filters size 3,3 and relu
model1.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))
#add max pooling to reduce the size of output of first convolution layer
model1.add(MaxPooling2D(pool_size=(2, 2)))
#Convolution with 64 filters size 3,3 and relu
model1.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
#add max pooling to reduce the size of output of second convolution layer
model1.add(MaxPooling2D(pool_size=(2, 2)))
#reshape the output to a flat vector
model1.add(Flatten())
#fully connected layer with 128 neurons
model1.add(Dense(128, activation='relu'))
#add dropout to prevent overfitting
model1.add(Dropout(0.5))
#add output layer
model1.add(Dense(num_classes, activation='softmax'))
# Compile the model
model1.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model1.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
max_pooling2d (MaxPooling2D (None, 63, 63, 32) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 64) 18496
max_pooling2d_1 (MaxPooling (None, 30, 30, 64) 0
2D)
flatten (Flatten) (None, 57600) 0
dense (Dense) (None, 128) 7372928
dropout (Dropout) (None, 128) 0
dense_1 (Dense) (None, 12) 1548
=================================================================
Total params: 7,393,868
Trainable params: 7,393,868
Non-trainable params: 0
_________________________________________________________________
Model Visualization
#visualize the CNN architecture
plot_model(model1, to_file='model1_plot.png', show_shapes=True, show_layer_names=True)
#visualize the CNN architecture using visualkeras
visualkeras.layered_view(model1, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 60 epochs and validation_split as 10%
history1 = model1.fit(X_train,
y_train_onehot,
epochs=60,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/60 119/121 [============================>.] - ETA: 0s - loss: 1.9557 - accuracy: 0.3456 Epoch 1: val_accuracy improved from -inf to 0.39953, saving model to best_model.h5 121/121 [==============================] - 4s 24ms/step - loss: 1.9516 - accuracy: 0.3473 - val_loss: 1.6871 - val_accuracy: 0.3995 Epoch 2/60 118/121 [============================>.] - ETA: 0s - loss: 1.4500 - accuracy: 0.5085 Epoch 2: val_accuracy improved from 0.39953 to 0.57009, saving model to best_model.h5 121/121 [==============================] - 3s 24ms/step - loss: 1.4500 - accuracy: 0.5095 - val_loss: 1.2356 - val_accuracy: 0.5701 Epoch 3/60 118/121 [============================>.] - ETA: 0s - loss: 1.2128 - accuracy: 0.5879 Epoch 3: val_accuracy improved from 0.57009 to 0.59813, saving model to best_model.h5 121/121 [==============================] - 3s 24ms/step - loss: 1.2120 - accuracy: 0.5885 - val_loss: 1.2166 - val_accuracy: 0.5981 Epoch 4/60 118/121 [============================>.] - ETA: 0s - loss: 1.0462 - accuracy: 0.6496 Epoch 4: val_accuracy improved from 0.59813 to 0.63785, saving model to best_model.h5 121/121 [==============================] - 3s 23ms/step - loss: 1.0478 - accuracy: 0.6483 - val_loss: 1.0640 - val_accuracy: 0.6379 Epoch 5/60 118/121 [============================>.] - ETA: 0s - loss: 0.9670 - accuracy: 0.6700 Epoch 5: val_accuracy improved from 0.63785 to 0.66121, saving model to best_model.h5 121/121 [==============================] - 2s 21ms/step - loss: 0.9692 - accuracy: 0.6691 - val_loss: 1.0265 - val_accuracy: 0.6612 Epoch 6/60 121/121 [==============================] - ETA: 0s - loss: 0.8473 - accuracy: 0.7024 Epoch 6: val_accuracy improved from 0.66121 to 0.67523, saving model to best_model.h5 121/121 [==============================] - 5s 39ms/step - loss: 0.8473 - accuracy: 0.7024 - val_loss: 0.9550 - val_accuracy: 0.6752 Epoch 7/60 120/121 [============================>.] - ETA: 0s - loss: 0.7730 - accuracy: 0.7372 Epoch 7: val_accuracy did not improve from 0.67523 121/121 [==============================] - 2s 20ms/step - loss: 0.7724 - accuracy: 0.7375 - val_loss: 0.9919 - val_accuracy: 0.6589 Epoch 8/60 120/121 [============================>.] - ETA: 0s - loss: 0.6773 - accuracy: 0.7633 Epoch 8: val_accuracy improved from 0.67523 to 0.71729, saving model to best_model.h5 121/121 [==============================] - 3s 22ms/step - loss: 0.6779 - accuracy: 0.7627 - val_loss: 0.9073 - val_accuracy: 0.7173 Epoch 9/60 120/121 [============================>.] - ETA: 0s - loss: 0.5863 - accuracy: 0.7974 Epoch 9: val_accuracy did not improve from 0.71729 121/121 [==============================] - 2s 18ms/step - loss: 0.5861 - accuracy: 0.7975 - val_loss: 0.9788 - val_accuracy: 0.7033 Epoch 10/60 119/121 [============================>.] - ETA: 0s - loss: 0.5086 - accuracy: 0.8262 Epoch 10: val_accuracy improved from 0.71729 to 0.72196, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 0.5104 - accuracy: 0.8251 - val_loss: 0.9273 - val_accuracy: 0.7220 Epoch 11/60 119/121 [============================>.] - ETA: 0s - loss: 0.4790 - accuracy: 0.8290 Epoch 11: val_accuracy improved from 0.72196 to 0.72664, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 0.4803 - accuracy: 0.8290 - val_loss: 0.9444 - val_accuracy: 0.7266 Epoch 12/60 121/121 [==============================] - ETA: 0s - loss: 0.4102 - accuracy: 0.8568 Epoch 12: val_accuracy improved from 0.72664 to 0.73131, saving model to best_model.h5 121/121 [==============================] - 3s 22ms/step - loss: 0.4102 - accuracy: 0.8568 - val_loss: 0.9420 - val_accuracy: 0.7313 Epoch 13/60 118/121 [============================>.] - ETA: 0s - loss: 0.3798 - accuracy: 0.8673 Epoch 13: val_accuracy improved from 0.73131 to 0.74766, saving model to best_model.h5 121/121 [==============================] - 3s 23ms/step - loss: 0.3841 - accuracy: 0.8661 - val_loss: 0.9019 - val_accuracy: 0.7477 Epoch 14/60 121/121 [==============================] - ETA: 0s - loss: 0.3665 - accuracy: 0.8742 Epoch 14: val_accuracy improved from 0.74766 to 0.75234, saving model to best_model.h5 121/121 [==============================] - 3s 22ms/step - loss: 0.3665 - accuracy: 0.8742 - val_loss: 0.9895 - val_accuracy: 0.7523 Epoch 15/60 118/121 [============================>.] - ETA: 0s - loss: 0.3009 - accuracy: 0.8938 Epoch 15: val_accuracy did not improve from 0.75234 121/121 [==============================] - 2s 18ms/step - loss: 0.2996 - accuracy: 0.8945 - val_loss: 0.9900 - val_accuracy: 0.7407 Epoch 16/60 121/121 [==============================] - ETA: 0s - loss: 0.2900 - accuracy: 0.8997 Epoch 16: val_accuracy did not improve from 0.75234 121/121 [==============================] - 3s 21ms/step - loss: 0.2900 - accuracy: 0.8997 - val_loss: 0.9611 - val_accuracy: 0.7407 Epoch 17/60 120/121 [============================>.] - ETA: 0s - loss: 0.2824 - accuracy: 0.8979 Epoch 17: val_accuracy did not improve from 0.75234 121/121 [==============================] - 2s 18ms/step - loss: 0.2824 - accuracy: 0.8978 - val_loss: 1.0341 - val_accuracy: 0.7407 Epoch 18/60 120/121 [============================>.] - ETA: 0s - loss: 0.2479 - accuracy: 0.9146 Epoch 18: val_accuracy did not improve from 0.75234 121/121 [==============================] - 2s 19ms/step - loss: 0.2483 - accuracy: 0.9145 - val_loss: 1.1104 - val_accuracy: 0.7453 Epoch 19/60 120/121 [============================>.] - ETA: 0s - loss: 0.2387 - accuracy: 0.9128 Epoch 19: val_accuracy did not improve from 0.75234 121/121 [==============================] - 2s 20ms/step - loss: 0.2383 - accuracy: 0.9129 - val_loss: 1.0547 - val_accuracy: 0.7477 Epoch 20/60 118/121 [============================>.] - ETA: 0s - loss: 0.2062 - accuracy: 0.9266 Epoch 20: val_accuracy did not improve from 0.75234 121/121 [==============================] - 2s 20ms/step - loss: 0.2065 - accuracy: 0.9264 - val_loss: 1.0478 - val_accuracy: 0.7523 Epoch 21/60 119/121 [============================>.] - ETA: 0s - loss: 0.2306 - accuracy: 0.9160 Epoch 21: val_accuracy did not improve from 0.75234 121/121 [==============================] - 2s 18ms/step - loss: 0.2302 - accuracy: 0.9163 - val_loss: 1.2161 - val_accuracy: 0.7407 Epoch 22/60 119/121 [============================>.] - ETA: 0s - loss: 0.2000 - accuracy: 0.9280 Epoch 22: val_accuracy did not improve from 0.75234 121/121 [==============================] - 2s 18ms/step - loss: 0.2001 - accuracy: 0.9280 - val_loss: 1.1027 - val_accuracy: 0.7430 Epoch 23/60 118/121 [============================>.] - ETA: 0s - loss: 0.1923 - accuracy: 0.9301 Epoch 23: val_accuracy did not improve from 0.75234 121/121 [==============================] - 2s 18ms/step - loss: 0.1908 - accuracy: 0.9309 - val_loss: 1.1310 - val_accuracy: 0.7383 Epoch 23: early stopping
Notice that at Epoch 23 the training stopped since the validation loss had continued to increase during 10 epochs. The highest validation accuracy was 0.75 and validation loss of 1.1866.
Model1 - Accuracy vs Epoch curve
plt.plot(history1.history['accuracy'])
plt.plot(history1.history['val_accuracy'])
plt.title('model1 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
Notice that the model shows overfitting. Past epoch 12 the validation loss doesn't reduce or begins to increase.
The model performs poorly as we see that the training accuracy is high but the validation accuracy is poor, indicating that the model is overfitting.
Evaluation of model1 model.evaluate using the Test data
# Evaluate the model on the test data
score1 = model1.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score1[0])
print('Test accuracy:', score1[1])
Test loss: 1.2217235565185547 Test accuracy: 0.730526328086853
# Test Prediction
y_pred = model1.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 7ms/step
# Test Accuracy
accuracyScore1 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore1)
0.7305263157894737
Classification Report
# Compute the classification report
cr1 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr1)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.90 0.86 0.88 50
Fat Hen 0.75 0.75 0.75 48
Shepherds Purse 0.43 0.39 0.41 23
Common wheat 0.65 0.68 0.67 22
Common Chickweed 0.88 0.84 0.86 61
Charlock 0.77 0.92 0.84 39
Cleavers 0.83 0.69 0.75 29
Scentless Mayweed 0.64 0.75 0.69 52
Sugar beet 0.80 0.63 0.71 38
Maize 0.57 0.73 0.64 22
Black-grass 0.42 0.31 0.36 26
Loose Silky-bent 0.74 0.77 0.75 65
accuracy 0.73 475
macro avg 0.70 0.69 0.69 475
weighted avg 0.73 0.73 0.73 475
The model seems to perform well for classes such as Small-flowered Cranesbill, Common Chickweed, and Charlock with high precision, recall, and F1-score.
The model performs poorly with classes like Black-grass, Shepherds Purse, and Common wheat that have low precision, recall, and F1-score.
The macro-average and weighted-average F1-score are 0.69 and 0.73, which indicates that the model has an overall decent performance.
However, this model can be improved, especially for the clasess that perform poorly.
Scoresdf = Scoresdf.append({'model':'Model1','accuracy':accuracyScore1,'layers':'8','total parameters':'7,393,868', 'neurons':'128', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Set the confusion matrix
cm1 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm1, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Model1 one performs better than Model0 but the model isn't very good either. It trains 7,393,868 parameters which is about 4 times more parameters than Model0 and although this difference is massive, Model1 doesn't have a great performance either.
Model2 keeps growing, to try to improve on the previous model (Model1) we added more layers.
Layers : 10
Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu
Layer 9: Droput layer: 0.5
Optimizer: Adam
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
# Define the model architecture
model2 = Sequential()
#Convolution with 32 filters size 3,3 and relu
model2.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=(128,128,3)))
#add max pooling to reduce the size of output of first convolution layer
model2.add(MaxPooling2D(pool_size=(2,2)))
#Convolution with 64 filters size 3,3 and relu
model2.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
#add max pooling to reduce the size of output of first convolution layer
model2.add(MaxPooling2D(pool_size=(2,2)))
#Convolution with 128 filters size 3,3 and relu
model2.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
#add max pooling to reduce the size of output of first convolution layer
model2.add(MaxPooling2D(pool_size=(2,2)))
#reshape the output to a flat vector
model2.add(Flatten())
#fully connected layer with 128 neurons
model2.add(Dense(128, activation='relu'))
#add dropout to prevent overfitting
model2.add(Dropout(0.5))
#add output layer
model2.add(Dense(num_classes, activation='softmax'))
# Compile the model
model2.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model2.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
max_pooling2d (MaxPooling2D (None, 63, 63, 32) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 64) 18496
max_pooling2d_1 (MaxPooling (None, 30, 30, 64) 0
2D)
conv2d_2 (Conv2D) (None, 28, 28, 128) 73856
max_pooling2d_2 (MaxPooling (None, 14, 14, 128) 0
2D)
flatten (Flatten) (None, 25088) 0
dense (Dense) (None, 128) 3211392
dropout (Dropout) (None, 128) 0
dense_1 (Dense) (None, 12) 1548
=================================================================
Total params: 3,306,188
Trainable params: 3,306,188
Non-trainable params: 0
_________________________________________________________________
Model Visualization
#visualize the cnn model architecture
plot_model(model2, to_file='model2_plot.png', show_shapes=True, show_layer_names=True)
#visualize the cnn model architecture using visualkeras
visualkeras.layered_view(model2, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 40 epochs and validation_split as 10%
history2 = model2.fit(X_train,
y_train_onehot,
epochs=40,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/40 119/121 [============================>.] - ETA: 0s - loss: 1.9732 - accuracy: 0.3369 Epoch 1: val_accuracy improved from -inf to 0.44626, saving model to best_model.h5 121/121 [==============================] - 6s 25ms/step - loss: 1.9721 - accuracy: 0.3379 - val_loss: 1.5570 - val_accuracy: 0.4463 Epoch 2/40 118/121 [============================>.] - ETA: 0s - loss: 1.4752 - accuracy: 0.4923 Epoch 2: val_accuracy improved from 0.44626 to 0.52570, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 1.4720 - accuracy: 0.4939 - val_loss: 1.2610 - val_accuracy: 0.5257 Epoch 3/40 118/121 [============================>.] - ETA: 0s - loss: 1.2448 - accuracy: 0.5728 Epoch 3: val_accuracy improved from 0.52570 to 0.61916, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 1.2429 - accuracy: 0.5734 - val_loss: 1.0807 - val_accuracy: 0.6192 Epoch 4/40 118/121 [============================>.] - ETA: 0s - loss: 1.0740 - accuracy: 0.6382 Epoch 4: val_accuracy did not improve from 0.61916 121/121 [==============================] - 2s 19ms/step - loss: 1.0733 - accuracy: 0.6382 - val_loss: 1.0088 - val_accuracy: 0.6145 Epoch 5/40 119/121 [============================>.] - ETA: 0s - loss: 0.9800 - accuracy: 0.6594 Epoch 5: val_accuracy improved from 0.61916 to 0.69159, saving model to best_model.h5 121/121 [==============================] - 3s 22ms/step - loss: 0.9808 - accuracy: 0.6584 - val_loss: 0.8978 - val_accuracy: 0.6916 Epoch 6/40 120/121 [============================>.] - ETA: 0s - loss: 0.8985 - accuracy: 0.6867 Epoch 6: val_accuracy improved from 0.69159 to 0.74065, saving model to best_model.h5 121/121 [==============================] - 3s 22ms/step - loss: 0.8982 - accuracy: 0.6868 - val_loss: 0.8329 - val_accuracy: 0.7407 Epoch 7/40 118/121 [============================>.] - ETA: 0s - loss: 0.8048 - accuracy: 0.7259 Epoch 7: val_accuracy did not improve from 0.74065 121/121 [==============================] - 2s 20ms/step - loss: 0.8049 - accuracy: 0.7255 - val_loss: 0.8244 - val_accuracy: 0.7407 Epoch 8/40 118/121 [============================>.] - ETA: 0s - loss: 0.7244 - accuracy: 0.7407 Epoch 8: val_accuracy improved from 0.74065 to 0.76636, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 0.7228 - accuracy: 0.7406 - val_loss: 0.7660 - val_accuracy: 0.7664 Epoch 9/40 118/121 [============================>.] - ETA: 0s - loss: 0.6450 - accuracy: 0.7688 Epoch 9: val_accuracy improved from 0.76636 to 0.77103, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 0.6472 - accuracy: 0.7687 - val_loss: 0.7315 - val_accuracy: 0.7710 Epoch 10/40 118/121 [============================>.] - ETA: 0s - loss: 0.6278 - accuracy: 0.7773 Epoch 10: val_accuracy did not improve from 0.77103 121/121 [==============================] - 2s 20ms/step - loss: 0.6289 - accuracy: 0.7757 - val_loss: 0.7808 - val_accuracy: 0.7407 Epoch 11/40 119/121 [============================>.] - ETA: 0s - loss: 0.5677 - accuracy: 0.7946 Epoch 11: val_accuracy did not improve from 0.77103 121/121 [==============================] - 3s 22ms/step - loss: 0.5674 - accuracy: 0.7952 - val_loss: 0.7369 - val_accuracy: 0.7593 Epoch 12/40 120/121 [============================>.] - ETA: 0s - loss: 0.5108 - accuracy: 0.8104 Epoch 12: val_accuracy improved from 0.77103 to 0.78037, saving model to best_model.h5 121/121 [==============================] - 3s 23ms/step - loss: 0.5105 - accuracy: 0.8105 - val_loss: 0.6926 - val_accuracy: 0.7804 Epoch 13/40 121/121 [==============================] - ETA: 0s - loss: 0.4677 - accuracy: 0.8227 Epoch 13: val_accuracy improved from 0.78037 to 0.80140, saving model to best_model.h5 121/121 [==============================] - 3s 22ms/step - loss: 0.4677 - accuracy: 0.8227 - val_loss: 0.6959 - val_accuracy: 0.8014 Epoch 14/40 118/121 [============================>.] - ETA: 0s - loss: 0.4548 - accuracy: 0.8276 Epoch 14: val_accuracy did not improve from 0.80140 121/121 [==============================] - 2s 20ms/step - loss: 0.4534 - accuracy: 0.8290 - val_loss: 0.7239 - val_accuracy: 0.7897 Epoch 15/40 118/121 [============================>.] - ETA: 0s - loss: 0.4509 - accuracy: 0.8337 Epoch 15: val_accuracy did not improve from 0.80140 121/121 [==============================] - 2s 20ms/step - loss: 0.4492 - accuracy: 0.8349 - val_loss: 0.7702 - val_accuracy: 0.7850 Epoch 16/40 118/121 [============================>.] - ETA: 0s - loss: 0.4024 - accuracy: 0.8490 Epoch 16: val_accuracy did not improve from 0.80140 121/121 [==============================] - 2s 20ms/step - loss: 0.4057 - accuracy: 0.8479 - val_loss: 0.7375 - val_accuracy: 0.8014 Epoch 17/40 121/121 [==============================] - ETA: 0s - loss: 0.3776 - accuracy: 0.8589 Epoch 17: val_accuracy did not improve from 0.80140 121/121 [==============================] - 3s 22ms/step - loss: 0.3776 - accuracy: 0.8589 - val_loss: 0.7692 - val_accuracy: 0.7991 Epoch 18/40 118/121 [============================>.] - ETA: 0s - loss: 0.3786 - accuracy: 0.8594 Epoch 18: val_accuracy did not improve from 0.80140 121/121 [==============================] - 3s 21ms/step - loss: 0.3786 - accuracy: 0.8586 - val_loss: 0.8283 - val_accuracy: 0.7874 Epoch 19/40 120/121 [============================>.] - ETA: 0s - loss: 0.3526 - accuracy: 0.8641 Epoch 19: val_accuracy did not improve from 0.80140 121/121 [==============================] - 3s 21ms/step - loss: 0.3523 - accuracy: 0.8643 - val_loss: 0.7889 - val_accuracy: 0.7944 Epoch 20/40 119/121 [============================>.] - ETA: 0s - loss: 0.3298 - accuracy: 0.8737 Epoch 20: val_accuracy did not improve from 0.80140 121/121 [==============================] - 3s 27ms/step - loss: 0.3314 - accuracy: 0.8734 - val_loss: 0.8751 - val_accuracy: 0.7804 Epoch 21/40 118/121 [============================>.] - ETA: 0s - loss: 0.3306 - accuracy: 0.8718 Epoch 21: val_accuracy improved from 0.80140 to 0.80841, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 0.3314 - accuracy: 0.8711 - val_loss: 0.7938 - val_accuracy: 0.8084 Epoch 22/40 121/121 [==============================] - ETA: 0s - loss: 0.3051 - accuracy: 0.8807 Epoch 22: val_accuracy did not improve from 0.80841 121/121 [==============================] - 2s 20ms/step - loss: 0.3051 - accuracy: 0.8807 - val_loss: 0.7822 - val_accuracy: 0.7921 Epoch 22: early stopping
The training stopped at epoch 21 because the validation loss had increased during 10 epochs. The highest validation accuracy was 0.780 and validation loss 0.7628.
Model 2: Accuracy vs epochs plot
plt.plot(history2.history['accuracy'])
plt.plot(history2.history['val_accuracy'])
plt.title('model2 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
Notice that somewhere around epoch 15 the validation loss begins to increase.
Overall Model2 seems to perform better than the 2 previous models.
Evaluation of model2 model.evaluate using the Test data
# Evaluate the model on the test data
score2 = model2.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score2[0])
print('Test accuracy:', score2[1])
Test loss: 0.9905914068222046 Test accuracy: 0.7873684167861938
# Test Prediction
y_pred = model2.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 7ms/step
# Test Accuracy
accuracyScore2 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore2)
0.7873684210526316
Classification Report
# Compute the classification report
cr2 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr2)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.86 0.86 0.86 50
Fat Hen 0.75 0.88 0.81 48
Shepherds Purse 0.78 0.30 0.44 23
Common wheat 0.73 0.73 0.73 22
Common Chickweed 0.92 0.95 0.94 61
Charlock 0.90 0.92 0.91 39
Cleavers 0.81 0.86 0.83 29
Scentless Mayweed 0.70 0.75 0.72 52
Sugar beet 0.88 0.76 0.82 38
Maize 0.65 0.59 0.62 22
Black-grass 0.50 0.15 0.24 26
Loose Silky-bent 0.71 0.95 0.82 65
accuracy 0.79 475
macro avg 0.76 0.73 0.73 475
weighted avg 0.78 0.79 0.77 475
The overall accuracy of Model2 is 0.79, which is a ok. The precision and recall values for each class are also mostly good, with some classes having higher values than others.
The highest precision and recall values are achieved for "Small-flowered Cranesbill", "Fat Hen", "Common Chickweed", "Charlock", and "Cleavers", which means that the model can identify well these classes.
Some classes have lower precision and recall values, such as "Shepherds Purse", "Common wheat", "Scentless Mayweed", "Sugar beet", "Maize", and "Black-grass".
Scoresdf = Scoresdf.append({'model':'Model2','accuracy':accuracyScore2,'layers':'10','total parameters':'3,306,188', 'neurons':'128', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
#set and plot the Confusion matrix
cm2 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm2, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Model2 performs better than Model0 and Model1. It actually performs OK.
But we can improve the model for better performance.
To try to improve Model2, we change the optimizer from Adam to SGD.
Layers : 10
Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu
Layer 9: Droput layer: 0.5
Optimizer: SGD
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Lets used SGD as optimizer with the second model since we obtained a higher accuracy than the first model
# Using SGD Optimizer
sgd = SGD(learning_rate=0.01, momentum=0.9)
Model Implementation
# Define the model architecture
model3 = Sequential()
#Convolution with 32 filters size 3,3 and relu
model3.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=(128,128,3)))
#add max pooling to reduce the size of output of first convolution layer
model3.add(MaxPooling2D(pool_size=(2,2)))
#Convolution with 64 filters size 3,3 and relu
model3.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
#add max pooling to reduce the size of output of first convolution layer
model3.add(MaxPooling2D(pool_size=(2,2)))
#Convolution with 128 filters size 3,3 and relu
model3.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
#add max pooling to reduce the size of output of first convolution layer
model3.add(MaxPooling2D(pool_size=(2,2)))
#reshape the output to a flat vector
model3.add(Flatten())
#fully connected layer with 128 neurons
model3.add(Dense(128, activation='relu'))
#add dropout to prevent overfitting
model3.add(Dropout(0.5))
#add output layer
model3.add(Dense(num_classes, activation='softmax'))
# Compile the model
model3.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
# Print the model summary
model3.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
max_pooling2d (MaxPooling2D (None, 63, 63, 32) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 64) 18496
max_pooling2d_1 (MaxPooling (None, 30, 30, 64) 0
2D)
conv2d_2 (Conv2D) (None, 28, 28, 128) 73856
max_pooling2d_2 (MaxPooling (None, 14, 14, 128) 0
2D)
flatten (Flatten) (None, 25088) 0
dense (Dense) (None, 128) 3211392
dropout (Dropout) (None, 128) 0
dense_1 (Dense) (None, 12) 1548
=================================================================
Total params: 3,306,188
Trainable params: 3,306,188
Non-trainable params: 0
_________________________________________________________________
Model Visualization
#visualize the cnn model architecture
plot_model(model3, to_file='model3_plot.png', show_shapes=True, show_layer_names=True)
#visualize the cnn model architecture using visualkeras
visualkeras.layered_view(model3, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 40 epochs and validation_split as 10%
history3 = model3.fit(X_train, y_train_onehot, epochs=40, batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/40 121/121 [==============================] - ETA: 0s - loss: 2.3845 - accuracy: 0.1578 Epoch 1: val_accuracy improved from -inf to 0.32243, saving model to best_model.h5 121/121 [==============================] - 4s 26ms/step - loss: 2.3845 - accuracy: 0.1578 - val_loss: 2.0716 - val_accuracy: 0.3224 Epoch 2/40 118/121 [============================>.] - ETA: 0s - loss: 1.8602 - accuracy: 0.3665 Epoch 2: val_accuracy did not improve from 0.32243 121/121 [==============================] - 2s 20ms/step - loss: 1.8570 - accuracy: 0.3678 - val_loss: 2.1210 - val_accuracy: 0.2967 Epoch 3/40 118/121 [============================>.] - ETA: 0s - loss: 1.6527 - accuracy: 0.4256 Epoch 3: val_accuracy improved from 0.32243 to 0.45794, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 1.6484 - accuracy: 0.4273 - val_loss: 1.5749 - val_accuracy: 0.4579 Epoch 4/40 119/121 [============================>.] - ETA: 0s - loss: 1.4911 - accuracy: 0.4811 Epoch 4: val_accuracy did not improve from 0.45794 121/121 [==============================] - 2s 19ms/step - loss: 1.4852 - accuracy: 0.4830 - val_loss: 1.4450 - val_accuracy: 0.4579 Epoch 5/40 118/121 [============================>.] - ETA: 0s - loss: 1.3449 - accuracy: 0.5246 Epoch 5: val_accuracy improved from 0.45794 to 0.46028, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 1.3446 - accuracy: 0.5251 - val_loss: 1.6715 - val_accuracy: 0.4603 Epoch 6/40 118/121 [============================>.] - ETA: 0s - loss: 1.2553 - accuracy: 0.5662 Epoch 6: val_accuracy improved from 0.46028 to 0.50234, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 1.2546 - accuracy: 0.5667 - val_loss: 1.4218 - val_accuracy: 0.5023 Epoch 7/40 118/121 [============================>.] - ETA: 0s - loss: 1.1140 - accuracy: 0.6163 Epoch 7: val_accuracy improved from 0.50234 to 0.62850, saving model to best_model.h5 121/121 [==============================] - 3s 22ms/step - loss: 1.1108 - accuracy: 0.6168 - val_loss: 1.1198 - val_accuracy: 0.6285 Epoch 8/40 120/121 [============================>.] - ETA: 0s - loss: 1.0393 - accuracy: 0.6391 Epoch 8: val_accuracy improved from 0.62850 to 0.63551, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 1.0389 - accuracy: 0.6389 - val_loss: 1.0225 - val_accuracy: 0.6355 Epoch 9/40 119/121 [============================>.] - ETA: 0s - loss: 0.9215 - accuracy: 0.6786 Epoch 9: val_accuracy improved from 0.63551 to 0.69393, saving model to best_model.h5 121/121 [==============================] - 2s 21ms/step - loss: 0.9207 - accuracy: 0.6792 - val_loss: 0.9386 - val_accuracy: 0.6939 Epoch 10/40 118/121 [============================>.] - ETA: 0s - loss: 0.9077 - accuracy: 0.6851 Epoch 10: val_accuracy did not improve from 0.69393 121/121 [==============================] - 2s 20ms/step - loss: 0.9055 - accuracy: 0.6862 - val_loss: 0.9669 - val_accuracy: 0.6799 Epoch 11/40 118/121 [============================>.] - ETA: 0s - loss: 0.7901 - accuracy: 0.7219 Epoch 11: val_accuracy did not improve from 0.69393 121/121 [==============================] - 2s 20ms/step - loss: 0.7900 - accuracy: 0.7219 - val_loss: 1.0805 - val_accuracy: 0.6355 Epoch 12/40 119/121 [============================>.] - ETA: 0s - loss: 0.7174 - accuracy: 0.7453 Epoch 12: val_accuracy improved from 0.69393 to 0.73364, saving model to best_model.h5 121/121 [==============================] - 3s 21ms/step - loss: 0.7178 - accuracy: 0.7447 - val_loss: 0.8993 - val_accuracy: 0.7336 Epoch 13/40 119/121 [============================>.] - ETA: 0s - loss: 0.6675 - accuracy: 0.7655 Epoch 13: val_accuracy improved from 0.73364 to 0.73832, saving model to best_model.h5 121/121 [==============================] - 3s 22ms/step - loss: 0.6708 - accuracy: 0.7640 - val_loss: 0.8316 - val_accuracy: 0.7383 Epoch 14/40 119/121 [============================>.] - ETA: 0s - loss: 0.6881 - accuracy: 0.7595 Epoch 14: val_accuracy did not improve from 0.73832 121/121 [==============================] - 2s 20ms/step - loss: 0.6882 - accuracy: 0.7596 - val_loss: 0.8765 - val_accuracy: 0.7383 Epoch 15/40 118/121 [============================>.] - ETA: 0s - loss: 0.6067 - accuracy: 0.7844 Epoch 15: val_accuracy improved from 0.73832 to 0.79673, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 0.6055 - accuracy: 0.7848 - val_loss: 0.7043 - val_accuracy: 0.7967 Epoch 16/40 118/121 [============================>.] - ETA: 0s - loss: 0.5655 - accuracy: 0.7969 Epoch 16: val_accuracy did not improve from 0.79673 121/121 [==============================] - 2s 20ms/step - loss: 0.5661 - accuracy: 0.7962 - val_loss: 0.8609 - val_accuracy: 0.7500 Epoch 17/40 119/121 [============================>.] - ETA: 0s - loss: 0.4954 - accuracy: 0.8180 Epoch 17: val_accuracy did not improve from 0.79673 121/121 [==============================] - 2s 20ms/step - loss: 0.4939 - accuracy: 0.8186 - val_loss: 0.7640 - val_accuracy: 0.7593 Epoch 18/40 118/121 [============================>.] - ETA: 0s - loss: 0.4738 - accuracy: 0.8244 Epoch 18: val_accuracy did not improve from 0.79673 121/121 [==============================] - 3s 21ms/step - loss: 0.4720 - accuracy: 0.8251 - val_loss: 0.8498 - val_accuracy: 0.7734 Epoch 19/40 118/121 [============================>.] - ETA: 0s - loss: 0.4325 - accuracy: 0.8392 Epoch 19: val_accuracy did not improve from 0.79673 121/121 [==============================] - 3s 21ms/step - loss: 0.4319 - accuracy: 0.8391 - val_loss: 0.7769 - val_accuracy: 0.7710 Epoch 20/40 118/121 [============================>.] - ETA: 0s - loss: 0.4054 - accuracy: 0.8520 Epoch 20: val_accuracy did not improve from 0.79673 121/121 [==============================] - 2s 20ms/step - loss: 0.4040 - accuracy: 0.8531 - val_loss: 0.8047 - val_accuracy: 0.7780 Epoch 21/40 119/121 [============================>.] - ETA: 0s - loss: 0.3600 - accuracy: 0.8640 Epoch 21: val_accuracy improved from 0.79673 to 0.80140, saving model to best_model.h5 121/121 [==============================] - 2s 20ms/step - loss: 0.3609 - accuracy: 0.8648 - val_loss: 0.8184 - val_accuracy: 0.8014 Epoch 22/40 118/121 [============================>.] - ETA: 0s - loss: 0.3771 - accuracy: 0.8604 Epoch 22: val_accuracy did not improve from 0.80140 121/121 [==============================] - 2s 19ms/step - loss: 0.3747 - accuracy: 0.8612 - val_loss: 0.8573 - val_accuracy: 0.7477 Epoch 23/40 118/121 [============================>.] - ETA: 0s - loss: 0.3378 - accuracy: 0.8816 Epoch 23: val_accuracy did not improve from 0.80140 121/121 [==============================] - 2s 20ms/step - loss: 0.3357 - accuracy: 0.8825 - val_loss: 0.8310 - val_accuracy: 0.7640 Epoch 24/40 118/121 [============================>.] - ETA: 0s - loss: 0.3023 - accuracy: 0.8922 Epoch 24: val_accuracy did not improve from 0.80140 121/121 [==============================] - 3s 21ms/step - loss: 0.2997 - accuracy: 0.8932 - val_loss: 0.9448 - val_accuracy: 0.7593 Epoch 25/40 118/121 [============================>.] - ETA: 0s - loss: 0.3030 - accuracy: 0.8856 Epoch 25: val_accuracy did not improve from 0.80140 121/121 [==============================] - 3s 21ms/step - loss: 0.3038 - accuracy: 0.8856 - val_loss: 0.9751 - val_accuracy: 0.7827 Epoch 25: early stopping
Training stopped on epoch 28. The highest validation accuracy was 0.7921 where the validation loss was 0.844.
Model3 Accuracy vs Epoch curve
plt.plot(history3.history['accuracy'])
plt.plot(history3.history['val_accuracy'])
plt.title('model3 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
Around epoch 20, the validation accuracy is highest. The model seems to be performing decently around epoch 22. There is some overfitting in the data.
Evaluation of model3 model.evaluate using the Test data
# Evaluate the model on the test data
score3 = model3.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score3[0])
print('Test accuracy:', score3[1])
Test loss: 0.9461641907691956 Test accuracy: 0.7726315855979919
# Test Prediction
y_pred = model3.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 7ms/step
# Test Accuracy
accuracyScore3 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore3)
0.7726315789473684
Classification Report
# Compute the classification report
cr3 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr3)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.83 0.88 0.85 50
Fat Hen 0.83 0.81 0.82 48
Shepherds Purse 0.64 0.61 0.62 23
Common wheat 0.83 0.45 0.59 22
Common Chickweed 0.86 0.97 0.91 61
Charlock 0.86 0.92 0.89 39
Cleavers 0.91 0.72 0.81 29
Scentless Mayweed 0.75 0.79 0.77 52
Sugar beet 0.72 0.74 0.73 38
Maize 0.65 0.68 0.67 22
Black-grass 0.35 0.23 0.28 26
Loose Silky-bent 0.74 0.83 0.78 65
accuracy 0.77 475
macro avg 0.75 0.72 0.73 475
weighted avg 0.77 0.77 0.76 475
Model3s overall accuracy is at 0.76, and the macro average F1-score is at 0.73, while the weighted average is at 0.7.
The model performs well on some classes, such as Small-flowered Cranesbill, Fat Hen, Common Chickweed, and Charlock, but doesn't perform well with others like Black-grass and Shepherds Purse.
Scoresdf = Scoresdf.append({'model':'Model3','accuracy':accuracyScore3,'layers':'10','total parameters':'3,306,188', 'neurons':'128', 'optimizer':'SGD'},ignore_index=True)
Confusion Matrix
# Computer the Confusion matrix
cm3 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
sns.heatmap(cm3, xticklabels=name_classes, yticklabels=name_classes, annot=True)
<Axes: >
Overall, model2 performs better than model3.
Lets try adding more layers to try to improve the previous models.
Layers : 17
Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu
Layer 2: BatchNormalization()
Layer 3: MaxPooling2D layer
Layer 4: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
Layer 5: BatchNormalization()
Layer 6: MaxPooling2D layer
Layer 7: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu
Layer 8: BatchNormalization()
Layer 9: MaxPooling2D layer
Layer 7: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu
Layer 8: BatchNormalization()
Layer 9: MaxPooling2D layer
Layer 10: Flatten layer
Layer 11: Dense layer : 256 neurons, Relu
Layer 12: BatchNormalization
Layer 13: Droput layer: 0.5
Optimizer: Adam
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
# initialized a sequential model
model4 = Sequential()
model4.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model4.add(BatchNormalization())
model4.add(MaxPooling2D(pool_size=(2,2)))
model4.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
model4.add(BatchNormalization())
model4.add(MaxPooling2D(pool_size=(2,2)))
model4.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model4.add(BatchNormalization())
model4.add(MaxPooling2D(pool_size=(2,2)))
model4.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model4.add(BatchNormalization())
model4.add(MaxPooling2D(pool_size=(2,2)))
model4.add(Flatten())
model4.add(Dense(128, activation='relu'))
model4.add(BatchNormalization())
model4.add(Dropout(0.5))
model4.add(Dense(num_classes, activation='softmax'))
# Compile the model
model4.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model4.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
batch_normalization (BatchN (None, 126, 126, 32) 128
ormalization)
max_pooling2d (MaxPooling2D (None, 63, 63, 32) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 64) 18496
batch_normalization_1 (Batc (None, 61, 61, 64) 256
hNormalization)
max_pooling2d_1 (MaxPooling (None, 30, 30, 64) 0
2D)
conv2d_2 (Conv2D) (None, 28, 28, 128) 73856
batch_normalization_2 (Batc (None, 28, 28, 128) 512
hNormalization)
max_pooling2d_2 (MaxPooling (None, 14, 14, 128) 0
2D)
conv2d_3 (Conv2D) (None, 12, 12, 128) 147584
batch_normalization_3 (Batc (None, 12, 12, 128) 512
hNormalization)
max_pooling2d_3 (MaxPooling (None, 6, 6, 128) 0
2D)
flatten (Flatten) (None, 4608) 0
dense (Dense) (None, 128) 589952
batch_normalization_4 (Batc (None, 128) 512
hNormalization)
dropout (Dropout) (None, 128) 0
dense_1 (Dense) (None, 12) 1548
=================================================================
Total params: 834,252
Trainable params: 833,292
Non-trainable params: 960
_________________________________________________________________
Model Visualization
#visualize model4 architecture
plot_model(model4, to_file='model4_plot.png', show_shapes=True, show_layer_names=True)
#visualize model4 architecture using visualkeras
visualkeras.layered_view(model4, legend=True)
Early Stopping
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history4=model4.fit(X_train,
y_train_onehot,
epochs=100,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/100 121/121 [==============================] - ETA: 0s - loss: 1.6201 - accuracy: 0.5134 Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5 121/121 [==============================] - 9s 31ms/step - loss: 1.6201 - accuracy: 0.5134 - val_loss: 5.6225 - val_accuracy: 0.1215 Epoch 2/100 120/121 [============================>.] - ETA: 0s - loss: 0.8899 - accuracy: 0.7138 Epoch 2: val_accuracy did not improve from 0.12150 121/121 [==============================] - 3s 27ms/step - loss: 0.8915 - accuracy: 0.7135 - val_loss: 10.7526 - val_accuracy: 0.1215 Epoch 3/100 119/121 [============================>.] - ETA: 0s - loss: 0.6252 - accuracy: 0.7915 Epoch 3: val_accuracy improved from 0.12150 to 0.12383, saving model to best_model.h5 121/121 [==============================] - 3s 28ms/step - loss: 0.6285 - accuracy: 0.7902 - val_loss: 10.3552 - val_accuracy: 0.1238 Epoch 4/100 119/121 [============================>.] - ETA: 0s - loss: 0.5196 - accuracy: 0.8267 Epoch 4: val_accuracy improved from 0.12383 to 0.24299, saving model to best_model.h5 121/121 [==============================] - 4s 32ms/step - loss: 0.5195 - accuracy: 0.8269 - val_loss: 4.9083 - val_accuracy: 0.2430 Epoch 5/100 120/121 [============================>.] - ETA: 0s - loss: 0.4166 - accuracy: 0.8544 Epoch 5: val_accuracy improved from 0.24299 to 0.30374, saving model to best_model.h5 121/121 [==============================] - 3s 28ms/step - loss: 0.4166 - accuracy: 0.8547 - val_loss: 3.4493 - val_accuracy: 0.3037 Epoch 6/100 120/121 [============================>.] - ETA: 0s - loss: 0.3413 - accuracy: 0.8857 Epoch 6: val_accuracy improved from 0.30374 to 0.59579, saving model to best_model.h5 121/121 [==============================] - 3s 28ms/step - loss: 0.3429 - accuracy: 0.8854 - val_loss: 1.3685 - val_accuracy: 0.5958 Epoch 7/100 119/121 [============================>.] - ETA: 0s - loss: 0.2859 - accuracy: 0.9097 Epoch 7: val_accuracy improved from 0.59579 to 0.86682, saving model to best_model.h5 121/121 [==============================] - 3s 27ms/step - loss: 0.2853 - accuracy: 0.9098 - val_loss: 0.3992 - val_accuracy: 0.8668 Epoch 8/100 119/121 [============================>.] - ETA: 0s - loss: 0.2253 - accuracy: 0.9315 Epoch 8: val_accuracy did not improve from 0.86682 121/121 [==============================] - 4s 30ms/step - loss: 0.2259 - accuracy: 0.9306 - val_loss: 1.2766 - val_accuracy: 0.6332 Epoch 9/100 119/121 [============================>.] - ETA: 0s - loss: 0.1847 - accuracy: 0.9459 Epoch 9: val_accuracy did not improve from 0.86682 121/121 [==============================] - 3s 28ms/step - loss: 0.1850 - accuracy: 0.9459 - val_loss: 0.4840 - val_accuracy: 0.8248 Epoch 10/100 120/121 [============================>.] - ETA: 0s - loss: 0.1638 - accuracy: 0.9474 Epoch 10: val_accuracy did not improve from 0.86682 121/121 [==============================] - 3s 27ms/step - loss: 0.1647 - accuracy: 0.9470 - val_loss: 1.0341 - val_accuracy: 0.7009 Epoch 11/100 119/121 [============================>.] - ETA: 0s - loss: 0.1588 - accuracy: 0.9451 Epoch 11: val_accuracy did not improve from 0.86682 121/121 [==============================] - 3s 27ms/step - loss: 0.1594 - accuracy: 0.9449 - val_loss: 1.2241 - val_accuracy: 0.6799 Epoch 12/100 120/121 [============================>.] - ETA: 0s - loss: 0.1690 - accuracy: 0.9451 Epoch 12: val_accuracy did not improve from 0.86682 121/121 [==============================] - 3s 28ms/step - loss: 0.1699 - accuracy: 0.9444 - val_loss: 1.9335 - val_accuracy: 0.5818 Epoch 13/100 121/121 [==============================] - ETA: 0s - loss: 0.1456 - accuracy: 0.9485 Epoch 13: val_accuracy did not improve from 0.86682 121/121 [==============================] - 3s 29ms/step - loss: 0.1456 - accuracy: 0.9485 - val_loss: 0.7255 - val_accuracy: 0.7687 Epoch 14/100 120/121 [============================>.] - ETA: 0s - loss: 0.1349 - accuracy: 0.9568 Epoch 14: val_accuracy improved from 0.86682 to 0.86916, saving model to best_model.h5 121/121 [==============================] - 3s 27ms/step - loss: 0.1356 - accuracy: 0.9563 - val_loss: 0.4801 - val_accuracy: 0.8692 Epoch 15/100 120/121 [============================>.] - ETA: 0s - loss: 0.1090 - accuracy: 0.9648 Epoch 15: val_accuracy did not improve from 0.86916 121/121 [==============================] - 3s 27ms/step - loss: 0.1103 - accuracy: 0.9646 - val_loss: 0.5455 - val_accuracy: 0.8178 Epoch 16/100 120/121 [============================>.] - ETA: 0s - loss: 0.0723 - accuracy: 0.9812 Epoch 16: val_accuracy did not improve from 0.86916 121/121 [==============================] - 3s 27ms/step - loss: 0.0723 - accuracy: 0.9813 - val_loss: 0.6476 - val_accuracy: 0.7944 Epoch 17/100 119/121 [============================>.] - ETA: 0s - loss: 0.0633 - accuracy: 0.9819 Epoch 17: val_accuracy did not improve from 0.86916 121/121 [==============================] - 3s 29ms/step - loss: 0.0629 - accuracy: 0.9821 - val_loss: 1.0993 - val_accuracy: 0.7243 Epoch 17: early stopping
Training stopped on epoch 33. The highest validation accuracy was 0.8925 where the validation loss was 0.3573.
Model4 Accuracy vs Epoch curve
plt.plot(history4.history['accuracy'])
plt.plot(history4.history['val_accuracy'])
plt.title('model4 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
Notice that the model tends to overfit as epochs increased.
Evaluation of model4 model.evaluate using the Test data
# Evaluate the model on the test data
score4 = model4.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score4[0])
print('Test accuracy:', score4[1])
Test loss: 1.3278809785842896 Test accuracy: 0.7052631378173828
# Test Prediction
y_pred = model4.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 8ms/step
# Test Accuracy
accuracyScore4 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore4)
0.7052631578947368
Classification Report
# Compute the classification report
cr4 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr4)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.95 0.78 0.86 50
Fat Hen 0.47 0.98 0.64 48
Shepherds Purse 0.53 0.43 0.48 23
Common wheat 0.82 0.41 0.55 22
Common Chickweed 0.91 0.82 0.86 61
Charlock 1.00 0.82 0.90 39
Cleavers 0.96 0.86 0.91 29
Scentless Mayweed 1.00 0.33 0.49 52
Sugar beet 0.85 0.76 0.81 38
Maize 0.51 0.95 0.67 22
Black-grass 0.25 0.27 0.26 26
Loose Silky-bent 0.68 0.75 0.72 65
accuracy 0.71 475
macro avg 0.74 0.68 0.68 475
weighted avg 0.78 0.71 0.70 475
Model4 has an overall accuracy of 83%.
The precision scores range from 0.48 to 0.95, indicating that the model's ability to correctly identify positive instances varies by class. For example, the model has high precision in identifying Charlock and Cleavers, while it has relatively lower precision in identifying Black-grass and Shepherds Purse.
The recall scores range from 0.50 to 0.97, indicating that the model's ability to correctly identify positive instances also varies by class. For example, the model has high recall in identifying Common Chickweed and Charlock, while it has relatively lower recall in identifying Black-grass and Shepherds Purse.
The F1-scores are a harmonic mean of precision and recall and range from 0.49 to 0.92. The F1-score provides a balance between precision and recall, indicating the overall performance of the model for each class.
Scoresdf = Scoresdf.append({'model':'Model4','accuracy':accuracyScore4,'layers':'17','total parameters':'834,252', 'neurons':'128', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Compute Confusion matrix
cm4 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm4, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Model4 seems to perform well overall, but there is some variation in the performance of the model for different classes.
To continue trying to improve the previous model, we add more layers to our model.
Layers : 19
Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu
Layer 2: BatchNormalization()
Layer 3: MaxPooling2D layer
Layer 4: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
Layer 5: BatchNormalization()
Layer 6: MaxPooling2D layer
Layer 7: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu
Layer 8: BatchNormalization()
Layer 9: MaxPooling2D layer
Layer 10: Convolutional layer : 256 filters, Kernel_size = (3,3), Relu
Layer 11: BatchNormalization()
Layer 12: MaxPooling2D layer
Layer 13: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu
Layer 14: BatchNormalization()
Layer 15: MaxPooling2D layer
Layer 16: Flatten layer
Layer 17: Dense layer : 256 neurons, Relu
Layer 18: BatchNormalization
Layer 19: Droput layer: 0.5
Optimizer: Adam
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
# initialized a sequential model
model5 = Sequential()
model5.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))
model5.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))
model5.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))
model5.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))
model5.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))
model5.add(Flatten())
model5.add(Dense(256, activation='relu'))
model5.add(BatchNormalization())
model5.add(Dropout(0.5))
model5.add(Dense(num_classes, activation='softmax'))
# Compile the model
model5.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model5.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
batch_normalization (BatchN (None, 126, 126, 32) 128
ormalization)
max_pooling2d (MaxPooling2D (None, 63, 63, 32) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 64) 18496
batch_normalization_1 (Batc (None, 61, 61, 64) 256
hNormalization)
max_pooling2d_1 (MaxPooling (None, 30, 30, 64) 0
2D)
conv2d_2 (Conv2D) (None, 28, 28, 128) 73856
batch_normalization_2 (Batc (None, 28, 28, 128) 512
hNormalization)
max_pooling2d_2 (MaxPooling (None, 14, 14, 128) 0
2D)
conv2d_3 (Conv2D) (None, 12, 12, 256) 295168
batch_normalization_3 (Batc (None, 12, 12, 256) 1024
hNormalization)
max_pooling2d_3 (MaxPooling (None, 6, 6, 256) 0
2D)
conv2d_4 (Conv2D) (None, 4, 4, 128) 295040
batch_normalization_4 (Batc (None, 4, 4, 128) 512
hNormalization)
max_pooling2d_4 (MaxPooling (None, 2, 2, 128) 0
2D)
flatten (Flatten) (None, 512) 0
dense (Dense) (None, 256) 131328
batch_normalization_5 (Batc (None, 256) 1024
hNormalization)
dropout (Dropout) (None, 256) 0
dense_1 (Dense) (None, 12) 3084
=================================================================
Total params: 821,324
Trainable params: 819,596
Non-trainable params: 1,728
_________________________________________________________________
Model5 Architecture Visualization
##visualize model architecture
plot_model(model5, to_file='model5_plot.png', show_shapes=True, show_layer_names=True)
##model architecture visualization using visualkeras
visualkeras.layered_view(model5, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=5)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history5 = model5.fit(X_train,
y_train_onehot,
epochs=100,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/100 119/121 [============================>.] - ETA: 0s - loss: 1.6716 - accuracy: 0.5011 Epoch 1: val_accuracy improved from -inf to 0.06308, saving model to best_model.h5 121/121 [==============================] - 9s 36ms/step - loss: 1.6664 - accuracy: 0.5032 - val_loss: 7.5329 - val_accuracy: 0.0631 Epoch 2/100 119/121 [============================>.] - ETA: 0s - loss: 0.8780 - accuracy: 0.7124 Epoch 2: val_accuracy improved from 0.06308 to 0.12150, saving model to best_model.h5 121/121 [==============================] - 4s 32ms/step - loss: 0.8768 - accuracy: 0.7130 - val_loss: 13.4861 - val_accuracy: 0.1215 Epoch 3/100 119/121 [============================>.] - ETA: 0s - loss: 0.6572 - accuracy: 0.7723 Epoch 3: val_accuracy improved from 0.12150 to 0.14486, saving model to best_model.h5 121/121 [==============================] - 4s 30ms/step - loss: 0.6564 - accuracy: 0.7723 - val_loss: 12.1352 - val_accuracy: 0.1449 Epoch 4/100 120/121 [============================>.] - ETA: 0s - loss: 0.4920 - accuracy: 0.8268 Epoch 4: val_accuracy improved from 0.14486 to 0.15888, saving model to best_model.h5 121/121 [==============================] - 4s 29ms/step - loss: 0.4933 - accuracy: 0.8264 - val_loss: 7.2529 - val_accuracy: 0.1589 Epoch 5/100 119/121 [============================>.] - ETA: 0s - loss: 0.4023 - accuracy: 0.8595 Epoch 5: val_accuracy improved from 0.15888 to 0.54907, saving model to best_model.h5 121/121 [==============================] - 4s 32ms/step - loss: 0.4035 - accuracy: 0.8594 - val_loss: 1.6262 - val_accuracy: 0.5491 Epoch 6/100 120/121 [============================>.] - ETA: 0s - loss: 0.3601 - accuracy: 0.8763 Epoch 6: val_accuracy improved from 0.54907 to 0.72664, saving model to best_model.h5 121/121 [==============================] - 4s 31ms/step - loss: 0.3619 - accuracy: 0.8760 - val_loss: 0.8660 - val_accuracy: 0.7266 Epoch 7/100 120/121 [============================>.] - ETA: 0s - loss: 0.2981 - accuracy: 0.8898 Epoch 7: val_accuracy did not improve from 0.72664 121/121 [==============================] - 3s 28ms/step - loss: 0.2979 - accuracy: 0.8900 - val_loss: 0.9416 - val_accuracy: 0.6963 Epoch 8/100 119/121 [============================>.] - ETA: 0s - loss: 0.2452 - accuracy: 0.9136 Epoch 8: val_accuracy improved from 0.72664 to 0.78271, saving model to best_model.h5 121/121 [==============================] - 3s 29ms/step - loss: 0.2483 - accuracy: 0.9127 - val_loss: 0.6978 - val_accuracy: 0.7827 Epoch 9/100 119/121 [============================>.] - ETA: 0s - loss: 0.2397 - accuracy: 0.9107 Epoch 9: val_accuracy did not improve from 0.78271 121/121 [==============================] - 4s 30ms/step - loss: 0.2394 - accuracy: 0.9108 - val_loss: 1.1052 - val_accuracy: 0.6776 Epoch 10/100 119/121 [============================>.] - ETA: 0s - loss: 0.1977 - accuracy: 0.9244 Epoch 10: val_accuracy improved from 0.78271 to 0.81542, saving model to best_model.h5 121/121 [==============================] - 4s 33ms/step - loss: 0.1987 - accuracy: 0.9236 - val_loss: 0.5499 - val_accuracy: 0.8154 Epoch 11/100 119/121 [============================>.] - ETA: 0s - loss: 0.1850 - accuracy: 0.9288 Epoch 11: val_accuracy improved from 0.81542 to 0.83879, saving model to best_model.h5 121/121 [==============================] - 4s 29ms/step - loss: 0.1846 - accuracy: 0.9288 - val_loss: 0.5094 - val_accuracy: 0.8388 Epoch 12/100 119/121 [============================>.] - ETA: 0s - loss: 0.1423 - accuracy: 0.9533 Epoch 12: val_accuracy improved from 0.83879 to 0.84112, saving model to best_model.h5 121/121 [==============================] - 3s 29ms/step - loss: 0.1429 - accuracy: 0.9530 - val_loss: 0.5343 - val_accuracy: 0.8411 Epoch 13/100 120/121 [============================>.] - ETA: 0s - loss: 0.1648 - accuracy: 0.9406 Epoch 13: val_accuracy did not improve from 0.84112 121/121 [==============================] - 4s 30ms/step - loss: 0.1648 - accuracy: 0.9407 - val_loss: 0.7414 - val_accuracy: 0.7804 Epoch 14/100 119/121 [============================>.] - ETA: 0s - loss: 0.1324 - accuracy: 0.9554 Epoch 14: val_accuracy did not improve from 0.84112 121/121 [==============================] - 4s 31ms/step - loss: 0.1320 - accuracy: 0.9558 - val_loss: 0.7250 - val_accuracy: 0.8107 Epoch 15/100 119/121 [============================>.] - ETA: 0s - loss: 0.1177 - accuracy: 0.9624 Epoch 15: val_accuracy did not improve from 0.84112 121/121 [==============================] - 3s 28ms/step - loss: 0.1198 - accuracy: 0.9618 - val_loss: 0.7258 - val_accuracy: 0.7944 Epoch 16/100 120/121 [============================>.] - ETA: 0s - loss: 0.1284 - accuracy: 0.9544 Epoch 16: val_accuracy did not improve from 0.84112 121/121 [==============================] - 3s 28ms/step - loss: 0.1285 - accuracy: 0.9543 - val_loss: 0.4891 - val_accuracy: 0.8411 Epoch 17/100 120/121 [============================>.] - ETA: 0s - loss: 0.1123 - accuracy: 0.9620 Epoch 17: val_accuracy did not improve from 0.84112 121/121 [==============================] - 3s 28ms/step - loss: 0.1122 - accuracy: 0.9620 - val_loss: 0.6232 - val_accuracy: 0.8014 Epoch 18/100 119/121 [============================>.] - ETA: 0s - loss: 0.0725 - accuracy: 0.9764 Epoch 18: val_accuracy improved from 0.84112 to 0.85748, saving model to best_model.h5 121/121 [==============================] - 4s 32ms/step - loss: 0.0727 - accuracy: 0.9763 - val_loss: 0.4822 - val_accuracy: 0.8575 Epoch 19/100 119/121 [============================>.] - ETA: 0s - loss: 0.0758 - accuracy: 0.9751 Epoch 19: val_accuracy did not improve from 0.85748 121/121 [==============================] - 4s 30ms/step - loss: 0.0755 - accuracy: 0.9750 - val_loss: 0.4922 - val_accuracy: 0.8458 Epoch 20/100 120/121 [============================>.] - ETA: 0s - loss: 0.0633 - accuracy: 0.9794 Epoch 20: val_accuracy did not improve from 0.85748 121/121 [==============================] - 3s 28ms/step - loss: 0.0646 - accuracy: 0.9789 - val_loss: 1.2289 - val_accuracy: 0.6822 Epoch 21/100 120/121 [============================>.] - ETA: 0s - loss: 0.1150 - accuracy: 0.9578 Epoch 21: val_accuracy did not improve from 0.85748 121/121 [==============================] - 3s 28ms/step - loss: 0.1156 - accuracy: 0.9576 - val_loss: 0.6274 - val_accuracy: 0.8201 Epoch 22/100 119/121 [============================>.] - ETA: 0s - loss: 0.0844 - accuracy: 0.9688 Epoch 22: val_accuracy did not improve from 0.85748 121/121 [==============================] - 4s 30ms/step - loss: 0.0837 - accuracy: 0.9691 - val_loss: 1.7372 - val_accuracy: 0.6986 Epoch 23/100 119/121 [============================>.] - ETA: 0s - loss: 0.0455 - accuracy: 0.9863 Epoch 23: val_accuracy improved from 0.85748 to 0.87617, saving model to best_model.h5 121/121 [==============================] - 4s 30ms/step - loss: 0.0454 - accuracy: 0.9865 - val_loss: 0.4484 - val_accuracy: 0.8762 Epoch 24/100 120/121 [============================>.] - ETA: 0s - loss: 0.0505 - accuracy: 0.9854 Epoch 24: val_accuracy improved from 0.87617 to 0.87850, saving model to best_model.h5 121/121 [==============================] - 4s 29ms/step - loss: 0.0504 - accuracy: 0.9854 - val_loss: 0.3818 - val_accuracy: 0.8785 Epoch 25/100 120/121 [============================>.] - ETA: 0s - loss: 0.0419 - accuracy: 0.9878 Epoch 25: val_accuracy did not improve from 0.87850 121/121 [==============================] - 3s 28ms/step - loss: 0.0423 - accuracy: 0.9875 - val_loss: 0.7614 - val_accuracy: 0.8248 Epoch 26/100 119/121 [============================>.] - ETA: 0s - loss: 0.0427 - accuracy: 0.9856 Epoch 26: val_accuracy did not improve from 0.87850 121/121 [==============================] - 4s 31ms/step - loss: 0.0425 - accuracy: 0.9854 - val_loss: 0.8043 - val_accuracy: 0.8271 Epoch 27/100 120/121 [============================>.] - ETA: 0s - loss: 0.0470 - accuracy: 0.9844 Epoch 27: val_accuracy did not improve from 0.87850 121/121 [==============================] - 4s 31ms/step - loss: 0.0473 - accuracy: 0.9841 - val_loss: 3.3697 - val_accuracy: 0.5093 Epoch 28/100 120/121 [============================>.] - ETA: 0s - loss: 0.0518 - accuracy: 0.9844 Epoch 28: val_accuracy did not improve from 0.87850 121/121 [==============================] - 3s 28ms/step - loss: 0.0518 - accuracy: 0.9844 - val_loss: 0.5574 - val_accuracy: 0.8435 Epoch 29/100 120/121 [============================>.] - ETA: 0s - loss: 0.0408 - accuracy: 0.9875 Epoch 29: val_accuracy did not improve from 0.87850 121/121 [==============================] - 3s 28ms/step - loss: 0.0407 - accuracy: 0.9875 - val_loss: 1.0695 - val_accuracy: 0.7734 Epoch 29: early stopping
Training stopped at epoch 15. The highest validation accuracy was 0.8575 and validation loss is 0.4383.
Accuracy vs Epoch curve
plt.plot(history5.history['accuracy'])
plt.plot(history5.history['val_accuracy'])
plt.title('model5 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
The model is overfitting. However, around epoch 9, it has a high validation accuracy and a high training accuracy.
Evaluation of model5 model.evaluate using the Test data
# Evaluate the model on the test data
score5 = model5.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score5[0])
print('Test accuracy:', score5[1])
Test loss: 1.1474685668945312 Test accuracy: 0.7557894587516785
# Test Prediction
y_pred = model5.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 11ms/step
# Test Accuracy
accuracyScore5 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore5)
0.7557894736842106
Classification Report
# Compute the classification report
cr5 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr5)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.94 0.92 0.93 50
Fat Hen 0.95 0.85 0.90 48
Shepherds Purse 0.81 0.74 0.77 23
Common wheat 0.82 0.64 0.72 22
Common Chickweed 0.98 0.82 0.89 61
Charlock 0.97 0.90 0.93 39
Cleavers 0.93 0.86 0.89 29
Scentless Mayweed 0.44 0.92 0.60 52
Sugar beet 1.00 0.21 0.35 38
Maize 0.53 0.77 0.63 22
Black-grass 0.64 0.27 0.38 26
Loose Silky-bent 0.71 0.78 0.74 65
accuracy 0.76 475
macro avg 0.81 0.72 0.73 475
weighted avg 0.82 0.76 0.75 475
The accuracy of the model5 is 83%, and the weighted average F1-score is 0.82. The model performs relatively well in predicting most of the plant species, with high precision, recall and f1-scores, except for Black-grass.
However Model4 seems to be performing slightly better.
Scoresdf = Scoresdf.append({'model':'Model5','accuracy':accuracyScore5,'layers':'20','total parameters':'821,324', 'neurons':'256', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
#Visualize the Confusion matrix
cm5 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm5, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Model5 performs well for most classes but Black-grass.
To try to improve the performance of our previous models, we changethe layers and filters.
Layers: 16
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
# initialized a sequential model
model6 = Sequential()
model6.add(Conv2D(64, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model6.add(BatchNormalization())
model6.add(MaxPooling2D(pool_size=(2,2)))
model6.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model6.add(BatchNormalization())
model6.add(MaxPooling2D(pool_size=(2,2)))
model6.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
model6.add(BatchNormalization())
model6.add(MaxPooling2D(pool_size=(2,2)))
model6.add(Flatten())
model6.add(Dense(512, activation='relu'))
model6.add(BatchNormalization())
model6.add(Dropout(0.5))
model6.add(Dense(256, activation='relu'))
model6.add(BatchNormalization())
model6.add(Dropout(0.5))
model6.add(Dense(num_classes, activation='softmax'))
# Compile the model
model6.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model6.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 64) 1792
batch_normalization (BatchN (None, 126, 126, 64) 256
ormalization)
max_pooling2d (MaxPooling2D (None, 63, 63, 64) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 128) 73856
batch_normalization_1 (Batc (None, 61, 61, 128) 512
hNormalization)
max_pooling2d_1 (MaxPooling (None, 30, 30, 128) 0
2D)
conv2d_2 (Conv2D) (None, 28, 28, 256) 295168
batch_normalization_2 (Batc (None, 28, 28, 256) 1024
hNormalization)
max_pooling2d_2 (MaxPooling (None, 14, 14, 256) 0
2D)
flatten (Flatten) (None, 50176) 0
dense (Dense) (None, 512) 25690624
batch_normalization_3 (Batc (None, 512) 2048
hNormalization)
dropout (Dropout) (None, 512) 0
dense_1 (Dense) (None, 256) 131328
batch_normalization_4 (Batc (None, 256) 1024
hNormalization)
dropout_1 (Dropout) (None, 256) 0
dense_2 (Dense) (None, 12) 3084
=================================================================
Total params: 26,200,716
Trainable params: 26,198,284
Non-trainable params: 2,432
_________________________________________________________________
Model6 Architecture Visualization
#visualize model6 architecture
plot_model(model6, to_file='model6_plot.png', show_shapes=True, show_layer_names=True)
#visualize model6 architecture using visualkeras
visualkeras.layered_view(model6, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 40 epochs and validation_split as 10%
history6=model6.fit(X_train,
y_train_onehot,
epochs=40,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/40 120/121 [============================>.] - ETA: 0s - loss: 1.9669 - accuracy: 0.4292 Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5 121/121 [==============================] - 18s 98ms/step - loss: 1.9666 - accuracy: 0.4292 - val_loss: 7.0356 - val_accuracy: 0.1215 Epoch 2/40 120/121 [============================>.] - ETA: 0s - loss: 1.1810 - accuracy: 0.6206 Epoch 2: val_accuracy did not improve from 0.12150 121/121 [==============================] - 6s 53ms/step - loss: 1.1832 - accuracy: 0.6200 - val_loss: 8.4802 - val_accuracy: 0.1215 Epoch 3/40 120/121 [============================>.] - ETA: 0s - loss: 0.8575 - accuracy: 0.7164 Epoch 3: val_accuracy improved from 0.12150 to 0.12383, saving model to best_model.h5 121/121 [==============================] - 12s 99ms/step - loss: 0.8580 - accuracy: 0.7161 - val_loss: 6.8681 - val_accuracy: 0.1238 Epoch 4/40 121/121 [==============================] - ETA: 0s - loss: 0.6646 - accuracy: 0.7720 Epoch 4: val_accuracy improved from 0.12383 to 0.20794, saving model to best_model.h5 121/121 [==============================] - 13s 109ms/step - loss: 0.6646 - accuracy: 0.7720 - val_loss: 5.8048 - val_accuracy: 0.2079 Epoch 5/40 121/121 [==============================] - ETA: 0s - loss: 0.5248 - accuracy: 0.8131 Epoch 5: val_accuracy improved from 0.20794 to 0.42290, saving model to best_model.h5 121/121 [==============================] - 12s 102ms/step - loss: 0.5248 - accuracy: 0.8131 - val_loss: 2.2491 - val_accuracy: 0.4229 Epoch 6/40 121/121 [==============================] - ETA: 0s - loss: 0.4506 - accuracy: 0.8451 Epoch 6: val_accuracy improved from 0.42290 to 0.58879, saving model to best_model.h5 121/121 [==============================] - 12s 100ms/step - loss: 0.4506 - accuracy: 0.8451 - val_loss: 1.3419 - val_accuracy: 0.5888 Epoch 7/40 121/121 [==============================] - ETA: 0s - loss: 0.3584 - accuracy: 0.8713 Epoch 7: val_accuracy improved from 0.58879 to 0.62150, saving model to best_model.h5 121/121 [==============================] - 12s 101ms/step - loss: 0.3584 - accuracy: 0.8713 - val_loss: 1.1451 - val_accuracy: 0.6215 Epoch 8/40 120/121 [============================>.] - ETA: 0s - loss: 0.3424 - accuracy: 0.8823 Epoch 8: val_accuracy improved from 0.62150 to 0.76636, saving model to best_model.h5 121/121 [==============================] - 11s 94ms/step - loss: 0.3438 - accuracy: 0.8815 - val_loss: 0.7611 - val_accuracy: 0.7664 Epoch 9/40 121/121 [==============================] - ETA: 0s - loss: 0.3091 - accuracy: 0.8895 Epoch 9: val_accuracy did not improve from 0.76636 121/121 [==============================] - 6s 52ms/step - loss: 0.3091 - accuracy: 0.8895 - val_loss: 2.2853 - val_accuracy: 0.3715 Epoch 10/40 120/121 [============================>.] - ETA: 0s - loss: 0.3499 - accuracy: 0.8802 Epoch 10: val_accuracy did not improve from 0.76636 121/121 [==============================] - 6s 52ms/step - loss: 0.3516 - accuracy: 0.8799 - val_loss: 6.1112 - val_accuracy: 0.1706 Epoch 11/40 120/121 [============================>.] - ETA: 0s - loss: 0.2723 - accuracy: 0.9057 Epoch 11: val_accuracy did not improve from 0.76636 121/121 [==============================] - 6s 52ms/step - loss: 0.2727 - accuracy: 0.9054 - val_loss: 0.9662 - val_accuracy: 0.7173 Epoch 12/40 120/121 [============================>.] - ETA: 0s - loss: 0.1859 - accuracy: 0.9315 Epoch 12: val_accuracy improved from 0.76636 to 0.82944, saving model to best_model.h5 121/121 [==============================] - 9s 71ms/step - loss: 0.1868 - accuracy: 0.9314 - val_loss: 0.5295 - val_accuracy: 0.8294 Epoch 13/40 120/121 [============================>.] - ETA: 0s - loss: 0.1668 - accuracy: 0.9398 Epoch 13: val_accuracy did not improve from 0.82944 121/121 [==============================] - 6s 53ms/step - loss: 0.1668 - accuracy: 0.9400 - val_loss: 0.8630 - val_accuracy: 0.7617 Epoch 14/40 120/121 [============================>.] - ETA: 0s - loss: 0.1245 - accuracy: 0.9604 Epoch 14: val_accuracy did not improve from 0.82944 121/121 [==============================] - 6s 51ms/step - loss: 0.1244 - accuracy: 0.9605 - val_loss: 7.8960 - val_accuracy: 0.2243 Epoch 15/40 120/121 [============================>.] - ETA: 0s - loss: 0.1046 - accuracy: 0.9672 Epoch 15: val_accuracy did not improve from 0.82944 121/121 [==============================] - 6s 53ms/step - loss: 0.1080 - accuracy: 0.9670 - val_loss: 0.7944 - val_accuracy: 0.7640 Epoch 16/40 120/121 [============================>.] - ETA: 0s - loss: 0.0991 - accuracy: 0.9680 Epoch 16: val_accuracy did not improve from 0.82944 121/121 [==============================] - 6s 51ms/step - loss: 0.0990 - accuracy: 0.9680 - val_loss: 0.7597 - val_accuracy: 0.7780 Epoch 17/40 120/121 [============================>.] - ETA: 0s - loss: 0.0788 - accuracy: 0.9742 Epoch 17: val_accuracy did not improve from 0.82944 121/121 [==============================] - 6s 53ms/step - loss: 0.0787 - accuracy: 0.9743 - val_loss: 0.6409 - val_accuracy: 0.8224 Epoch 18/40 121/121 [==============================] - ETA: 0s - loss: 0.0746 - accuracy: 0.9727 Epoch 18: val_accuracy did not improve from 0.82944 121/121 [==============================] - 6s 53ms/step - loss: 0.0746 - accuracy: 0.9727 - val_loss: 0.9586 - val_accuracy: 0.7383 Epoch 19/40 120/121 [============================>.] - ETA: 0s - loss: 0.0653 - accuracy: 0.9773 Epoch 19: val_accuracy did not improve from 0.82944 121/121 [==============================] - 6s 52ms/step - loss: 0.0652 - accuracy: 0.9774 - val_loss: 1.3687 - val_accuracy: 0.6706 Epoch 20/40 121/121 [==============================] - ETA: 0s - loss: 0.0584 - accuracy: 0.9784 Epoch 20: val_accuracy did not improve from 0.82944 121/121 [==============================] - 6s 53ms/step - loss: 0.0584 - accuracy: 0.9784 - val_loss: 1.8634 - val_accuracy: 0.6565 Epoch 21/40 120/121 [============================>.] - ETA: 0s - loss: 0.1058 - accuracy: 0.9643 Epoch 21: val_accuracy did not improve from 0.82944 121/121 [==============================] - 6s 50ms/step - loss: 0.1058 - accuracy: 0.9644 - val_loss: 1.1467 - val_accuracy: 0.7290 Epoch 22/40 120/121 [============================>.] - ETA: 0s - loss: 0.0926 - accuracy: 0.9688 Epoch 22: val_accuracy did not improve from 0.82944 121/121 [==============================] - 6s 53ms/step - loss: 0.0926 - accuracy: 0.9688 - val_loss: 4.7129 - val_accuracy: 0.4650 Epoch 22: early stopping
Training stopped at epoch 24. The highest Validation accuracy was 0.8481 and validation loss 0.5729.
Model6 Accuracy vs Epoch curve
plt.plot(history6.history['accuracy'])
plt.plot(history6.history['val_accuracy'])
plt.title('model6 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
This model seems to be overfitting.
Evaluation of model6 model.evaluate using the Test data
# Evaluate the model on the test data
score6 = model6.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score6[0])
print('Test accuracy:', score6[1])
Test loss: 4.940274238586426 Test accuracy: 0.42105263471603394
# Test Prediction
y_pred = model6.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 13ms/step
# Test Accuracy
accuracyScore6 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore6)
0.42105263157894735
Classification Report
# Compute the classification report
cr6 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr6)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 1.00 0.46 0.63 50
Fat Hen 1.00 0.08 0.15 48
Shepherds Purse 0.57 0.17 0.27 23
Common wheat 0.00 0.00 0.00 22
Common Chickweed 0.94 0.56 0.70 61
Charlock 1.00 0.10 0.19 39
Cleavers 1.00 0.07 0.13 29
Scentless Mayweed 0.22 1.00 0.36 52
Sugar beet 1.00 0.03 0.05 38
Maize 0.33 0.64 0.43 22
Black-grass 0.50 0.12 0.19 26
Loose Silky-bent 0.54 0.91 0.67 65
accuracy 0.42 475
macro avg 0.67 0.34 0.31 475
weighted avg 0.72 0.42 0.37 475
The model seems to be overfitting as it has high accuracy on the training set but low accuracy on the test set. The precision, recall, and F1-score for each class also vary widely, with some classes having high scores and others having low scores. This suggests that the model is not performing well in some classes.
Scoresdf = Scoresdf.append({'model':'Model6','accuracy':accuracyScore6,'layers':'17','total parameters':'26,200,716', 'neurons':'256', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Calculate Confusion matrix
cm6 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm5, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Overall, Model6 doesn't perform well for some classes and is overfitting.
To continue trying to improve our model, we added more layers and modified the filter sizes.
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
#Model implementation
model7 = Sequential()
model7.add(Conv2D(32, (3, 3), activation='relu', input_shape=input_shape))
model7.add(Conv2D(32, (3, 3), activation='relu'))
model7.add(BatchNormalization())
model7.add(MaxPooling2D(pool_size=(2, 2)))
model7.add(Dropout(0.25))
model7.add(Conv2D(64, (3, 3), activation='relu'))
model7.add(Conv2D(64, (3, 3), activation='relu'))
model7.add(BatchNormalization())
model7.add(MaxPooling2D(pool_size=(2, 2)))
model7.add(Dropout(0.25))
model7.add(Conv2D(128, (3, 3), activation='relu'))
model7.add(Conv2D(128, (3, 3), activation='relu'))
model7.add(BatchNormalization())
model7.add(MaxPooling2D(pool_size=(2, 2)))
model7.add(Dropout(0.25))
model7.add(Flatten())
model7.add(Dense(512, activation='relu'))
model7.add(BatchNormalization())
model7.add(Dropout(0.5))
model7.add(Dense(num_classes, activation='softmax'))
model7.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model7.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 32) 896
conv2d_1 (Conv2D) (None, 124, 124, 32) 9248
batch_normalization (BatchN (None, 124, 124, 32) 128
ormalization)
max_pooling2d (MaxPooling2D (None, 62, 62, 32) 0
)
dropout (Dropout) (None, 62, 62, 32) 0
conv2d_2 (Conv2D) (None, 60, 60, 64) 18496
conv2d_3 (Conv2D) (None, 58, 58, 64) 36928
batch_normalization_1 (Batc (None, 58, 58, 64) 256
hNormalization)
max_pooling2d_1 (MaxPooling (None, 29, 29, 64) 0
2D)
dropout_1 (Dropout) (None, 29, 29, 64) 0
conv2d_4 (Conv2D) (None, 27, 27, 128) 73856
conv2d_5 (Conv2D) (None, 25, 25, 128) 147584
batch_normalization_2 (Batc (None, 25, 25, 128) 512
hNormalization)
max_pooling2d_2 (MaxPooling (None, 12, 12, 128) 0
2D)
dropout_2 (Dropout) (None, 12, 12, 128) 0
flatten (Flatten) (None, 18432) 0
dense (Dense) (None, 512) 9437696
batch_normalization_3 (Batc (None, 512) 2048
hNormalization)
dropout_3 (Dropout) (None, 512) 0
dense_1 (Dense) (None, 12) 6156
=================================================================
Total params: 9,733,804
Trainable params: 9,732,332
Non-trainable params: 1,472
_________________________________________________________________
Model7 Architecture Visualization
#visualize model7 architecture
plot_model(model7, to_file='model7_plot.png', show_shapes=True, show_layer_names=True)
#visualize model7 architecture using visualkeras
visualkeras.layered_view(model7, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history7=model7.fit(X_train,
y_train_onehot,
epochs=100,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/100 120/121 [============================>.] - ETA: 0s - loss: 1.9258 - accuracy: 0.4435 Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5 121/121 [==============================] - 14s 73ms/step - loss: 1.9249 - accuracy: 0.4440 - val_loss: 23.2398 - val_accuracy: 0.1215 Epoch 2/100 120/121 [============================>.] - ETA: 0s - loss: 1.0223 - accuracy: 0.6716 Epoch 2: val_accuracy did not improve from 0.12150 121/121 [==============================] - 8s 63ms/step - loss: 1.0225 - accuracy: 0.6712 - val_loss: 18.6191 - val_accuracy: 0.1215 Epoch 3/100 120/121 [============================>.] - ETA: 0s - loss: 0.6844 - accuracy: 0.7745 Epoch 3: val_accuracy did not improve from 0.12150 121/121 [==============================] - 8s 63ms/step - loss: 0.6857 - accuracy: 0.7744 - val_loss: 21.4383 - val_accuracy: 0.1215 Epoch 4/100 120/121 [============================>.] - ETA: 0s - loss: 0.5225 - accuracy: 0.8263 Epoch 4: val_accuracy improved from 0.12150 to 0.14252, saving model to best_model.h5 121/121 [==============================] - 8s 70ms/step - loss: 0.5232 - accuracy: 0.8261 - val_loss: 7.6242 - val_accuracy: 0.1425 Epoch 5/100 120/121 [============================>.] - ETA: 0s - loss: 0.4009 - accuracy: 0.8602 Epoch 5: val_accuracy improved from 0.14252 to 0.18925, saving model to best_model.h5 121/121 [==============================] - 8s 68ms/step - loss: 0.4030 - accuracy: 0.8594 - val_loss: 5.4088 - val_accuracy: 0.1893 Epoch 6/100 120/121 [============================>.] - ETA: 0s - loss: 0.3923 - accuracy: 0.8622 Epoch 6: val_accuracy improved from 0.18925 to 0.67290, saving model to best_model.h5 121/121 [==============================] - 14s 113ms/step - loss: 0.3940 - accuracy: 0.8615 - val_loss: 1.0581 - val_accuracy: 0.6729 Epoch 7/100 120/121 [============================>.] - ETA: 0s - loss: 0.3071 - accuracy: 0.8935 Epoch 7: val_accuracy improved from 0.67290 to 0.79673, saving model to best_model.h5 121/121 [==============================] - 8s 69ms/step - loss: 0.3068 - accuracy: 0.8937 - val_loss: 0.7298 - val_accuracy: 0.7967 Epoch 8/100 120/121 [============================>.] - ETA: 0s - loss: 0.2381 - accuracy: 0.9133 Epoch 8: val_accuracy did not improve from 0.79673 121/121 [==============================] - 8s 62ms/step - loss: 0.2386 - accuracy: 0.9132 - val_loss: 1.3537 - val_accuracy: 0.6612 Epoch 9/100 120/121 [============================>.] - ETA: 0s - loss: 0.1960 - accuracy: 0.9320 Epoch 9: val_accuracy improved from 0.79673 to 0.79907, saving model to best_model.h5 121/121 [==============================] - 8s 68ms/step - loss: 0.1963 - accuracy: 0.9319 - val_loss: 0.7284 - val_accuracy: 0.7991 Epoch 10/100 120/121 [============================>.] - ETA: 0s - loss: 0.1645 - accuracy: 0.9409 Epoch 10: val_accuracy did not improve from 0.79907 121/121 [==============================] - 8s 62ms/step - loss: 0.1655 - accuracy: 0.9402 - val_loss: 0.8174 - val_accuracy: 0.7617 Epoch 11/100 120/121 [============================>.] - ETA: 0s - loss: 0.1333 - accuracy: 0.9555 Epoch 11: val_accuracy improved from 0.79907 to 0.85047, saving model to best_model.h5 121/121 [==============================] - 8s 67ms/step - loss: 0.1351 - accuracy: 0.9550 - val_loss: 0.4578 - val_accuracy: 0.8505 Epoch 12/100 120/121 [============================>.] - ETA: 0s - loss: 0.1134 - accuracy: 0.9591 Epoch 12: val_accuracy did not improve from 0.85047 121/121 [==============================] - 8s 63ms/step - loss: 0.1148 - accuracy: 0.9584 - val_loss: 1.0042 - val_accuracy: 0.7593 Epoch 13/100 120/121 [============================>.] - ETA: 0s - loss: 0.1403 - accuracy: 0.9497 Epoch 13: val_accuracy did not improve from 0.85047 121/121 [==============================] - 8s 65ms/step - loss: 0.1414 - accuracy: 0.9493 - val_loss: 0.6063 - val_accuracy: 0.8458 Epoch 14/100 120/121 [============================>.] - ETA: 0s - loss: 0.1181 - accuracy: 0.9583 Epoch 14: val_accuracy did not improve from 0.85047 121/121 [==============================] - 8s 64ms/step - loss: 0.1183 - accuracy: 0.9581 - val_loss: 1.3537 - val_accuracy: 0.6682 Epoch 15/100 120/121 [============================>.] - ETA: 0s - loss: 0.1149 - accuracy: 0.9628 Epoch 15: val_accuracy did not improve from 0.85047 121/121 [==============================] - 8s 63ms/step - loss: 0.1179 - accuracy: 0.9623 - val_loss: 1.4002 - val_accuracy: 0.6799 Epoch 16/100 120/121 [============================>.] - ETA: 0s - loss: 0.0929 - accuracy: 0.9680 Epoch 16: val_accuracy did not improve from 0.85047 121/121 [==============================] - 8s 64ms/step - loss: 0.0929 - accuracy: 0.9680 - val_loss: 0.7927 - val_accuracy: 0.8201 Epoch 17/100 120/121 [============================>.] - ETA: 0s - loss: 0.0706 - accuracy: 0.9747 Epoch 17: val_accuracy improved from 0.85047 to 0.85280, saving model to best_model.h5 121/121 [==============================] - 9s 72ms/step - loss: 0.0705 - accuracy: 0.9748 - val_loss: 0.5961 - val_accuracy: 0.8528 Epoch 18/100 120/121 [============================>.] - ETA: 0s - loss: 0.0556 - accuracy: 0.9815 Epoch 18: val_accuracy did not improve from 0.85280 121/121 [==============================] - 8s 64ms/step - loss: 0.0561 - accuracy: 0.9813 - val_loss: 1.0609 - val_accuracy: 0.7313 Epoch 19/100 120/121 [============================>.] - ETA: 0s - loss: 0.1063 - accuracy: 0.9648 Epoch 19: val_accuracy did not improve from 0.85280 121/121 [==============================] - 8s 64ms/step - loss: 0.1063 - accuracy: 0.9649 - val_loss: 0.5733 - val_accuracy: 0.8458 Epoch 20/100 120/121 [============================>.] - ETA: 0s - loss: 0.0721 - accuracy: 0.9755 Epoch 20: val_accuracy did not improve from 0.85280 121/121 [==============================] - 8s 63ms/step - loss: 0.0728 - accuracy: 0.9753 - val_loss: 0.5768 - val_accuracy: 0.8528 Epoch 21/100 120/121 [============================>.] - ETA: 0s - loss: 0.0902 - accuracy: 0.9714 Epoch 21: val_accuracy did not improve from 0.85280 121/121 [==============================] - 8s 64ms/step - loss: 0.0900 - accuracy: 0.9714 - val_loss: 0.6353 - val_accuracy: 0.8248 Epoch 21: early stopping
Training stopped at epoch 27. The highest validation accuracy was 0.8578 with validation loss of 0.25
Model7 Accuracy vs Epoch curve
plt.plot(history7.history['accuracy'])
plt.plot(history7.history['val_accuracy'])
plt.title('model7 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
Model is overfitting
Model7 Evaluation of model model.evaluate using the Test data
# Evaluate the model on the test data
score7 = model7.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score7[0])
print('Test accuracy:', score7[1])
Test loss: 0.7704196572303772 Test accuracy: 0.7957894802093506
# Test Prediction
y_pred = model7.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 13ms/step
# Test Accuracy
accuracyScore7 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore7)
0.7957894736842105
Classification Report
# Compute the classification report
cr7 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr7)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.89 0.80 0.84 50
Fat Hen 0.75 0.96 0.84 48
Shepherds Purse 0.78 0.61 0.68 23
Common wheat 0.76 0.59 0.67 22
Common Chickweed 0.94 0.80 0.87 61
Charlock 0.84 0.95 0.89 39
Cleavers 0.87 0.90 0.88 29
Scentless Mayweed 0.85 0.87 0.86 52
Sugar beet 0.82 0.84 0.83 38
Maize 0.83 0.68 0.75 22
Black-grass 0.38 0.46 0.41 26
Loose Silky-bent 0.74 0.75 0.75 65
accuracy 0.80 475
macro avg 0.79 0.77 0.77 475
weighted avg 0.81 0.80 0.80 475
The accuracy of the model is very low at 0.24. For most of the classes, precision, recall, and f1-score are also very low. This suggests that the model is not able to classify the images correctly, and the predictions made by the model are almost random. Therefore, it is not a valid model.
Scoresdf = Scoresdf.append({'model':'Model7','accuracy':accuracyScore7,'layers':'20','total parameters':'9,733,804', 'neurons':'512', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Compute the Confusion matrix
cm7 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
sns.heatmap(cm7, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.show()
Model7 is not a valid model.
Continuing trying to improve the previous model, we changed the number of layers and neurons.
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
Model Implementation
# initialized a sequential model
model8 = Sequential()
model8.add(Conv2D(64, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))
model8.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))
model8.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))
model8.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))
model8.add(Conv2D(64, kernel_size=(5,5), activation='relu'))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))
model8.add(Flatten())
model8.add(Dense(256, activation='relu'))
model8.add(BatchNormalization())
model8.add(Dropout(0.5))
model8.add(Dense(num_classes, activation='softmax'))
# Compile the model
model8.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# Print the model summary
model8.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 126, 126, 64) 1792
batch_normalization (BatchN (None, 126, 126, 64) 256
ormalization)
max_pooling2d (MaxPooling2D (None, 63, 63, 64) 0
)
conv2d_1 (Conv2D) (None, 61, 61, 128) 73856
batch_normalization_1 (Batc (None, 61, 61, 128) 512
hNormalization)
max_pooling2d_1 (MaxPooling (None, 30, 30, 128) 0
2D)
conv2d_2 (Conv2D) (None, 28, 28, 256) 295168
batch_normalization_2 (Batc (None, 28, 28, 256) 1024
hNormalization)
max_pooling2d_2 (MaxPooling (None, 14, 14, 256) 0
2D)
conv2d_3 (Conv2D) (None, 12, 12, 128) 295040
batch_normalization_3 (Batc (None, 12, 12, 128) 512
hNormalization)
max_pooling2d_3 (MaxPooling (None, 6, 6, 128) 0
2D)
conv2d_4 (Conv2D) (None, 2, 2, 64) 204864
batch_normalization_4 (Batc (None, 2, 2, 64) 256
hNormalization)
max_pooling2d_4 (MaxPooling (None, 1, 1, 64) 0
2D)
flatten (Flatten) (None, 64) 0
dense (Dense) (None, 256) 16640
batch_normalization_5 (Batc (None, 256) 1024
hNormalization)
dropout (Dropout) (None, 256) 0
dense_1 (Dense) (None, 12) 3084
=================================================================
Total params: 894,028
Trainable params: 892,236
Non-trainable params: 1,792
_________________________________________________________________
Model8 Architecture Visualization
#visualize model4 architecture
plot_model(model8, to_file='model8_plot.png', show_shapes=True, show_layer_names=True)
#visualize model8 architecture using visualkeras
visualkeras.layered_view(model8, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history8=model8.fit(X_train,
y_train_onehot,
epochs=100,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/100 121/121 [==============================] - ETA: 0s - loss: 1.9141 - accuracy: 0.4328 Epoch 1: val_accuracy improved from -inf to 0.11916, saving model to best_model.h5 121/121 [==============================] - 12s 51ms/step - loss: 1.9141 - accuracy: 0.4328 - val_loss: 6.6545 - val_accuracy: 0.1192 Epoch 2/100 121/121 [==============================] - ETA: 0s - loss: 1.0103 - accuracy: 0.6740 Epoch 2: val_accuracy improved from 0.11916 to 0.12150, saving model to best_model.h5 121/121 [==============================] - 6s 49ms/step - loss: 1.0103 - accuracy: 0.6740 - val_loss: 13.4881 - val_accuracy: 0.1215 Epoch 3/100 120/121 [============================>.] - ETA: 0s - loss: 0.7239 - accuracy: 0.7560 Epoch 3: val_accuracy improved from 0.12150 to 0.12850, saving model to best_model.h5 121/121 [==============================] - 6s 49ms/step - loss: 0.7257 - accuracy: 0.7554 - val_loss: 9.6117 - val_accuracy: 0.1285 Epoch 4/100 120/121 [============================>.] - ETA: 0s - loss: 0.5796 - accuracy: 0.8023 Epoch 4: val_accuracy did not improve from 0.12850 121/121 [==============================] - 6s 48ms/step - loss: 0.5814 - accuracy: 0.8019 - val_loss: 8.5606 - val_accuracy: 0.1285 Epoch 5/100 120/121 [============================>.] - ETA: 0s - loss: 0.4688 - accuracy: 0.8372 Epoch 5: val_accuracy improved from 0.12850 to 0.43692, saving model to best_model.h5 121/121 [==============================] - 6s 49ms/step - loss: 0.4694 - accuracy: 0.8368 - val_loss: 2.2889 - val_accuracy: 0.4369 Epoch 6/100 121/121 [==============================] - ETA: 0s - loss: 0.4436 - accuracy: 0.8417 Epoch 6: val_accuracy improved from 0.43692 to 0.51636, saving model to best_model.h5 121/121 [==============================] - 6s 49ms/step - loss: 0.4436 - accuracy: 0.8417 - val_loss: 1.5959 - val_accuracy: 0.5164 Epoch 7/100 120/121 [============================>.] - ETA: 0s - loss: 0.3202 - accuracy: 0.8878 Epoch 7: val_accuracy improved from 0.51636 to 0.69860, saving model to best_model.h5 121/121 [==============================] - 6s 50ms/step - loss: 0.3200 - accuracy: 0.8880 - val_loss: 1.0497 - val_accuracy: 0.6986 Epoch 8/100 121/121 [==============================] - ETA: 0s - loss: 0.3005 - accuracy: 0.8942 Epoch 8: val_accuracy did not improve from 0.69860 121/121 [==============================] - 6s 48ms/step - loss: 0.3005 - accuracy: 0.8942 - val_loss: 5.7750 - val_accuracy: 0.2056 Epoch 9/100 120/121 [============================>.] - ETA: 0s - loss: 0.2668 - accuracy: 0.9023 Epoch 9: val_accuracy did not improve from 0.69860 121/121 [==============================] - 6s 49ms/step - loss: 0.2665 - accuracy: 0.9025 - val_loss: 7.3724 - val_accuracy: 0.2126 Epoch 10/100 121/121 [==============================] - ETA: 0s - loss: 0.2086 - accuracy: 0.9194 Epoch 10: val_accuracy improved from 0.69860 to 0.73598, saving model to best_model.h5 121/121 [==============================] - 6s 50ms/step - loss: 0.2086 - accuracy: 0.9194 - val_loss: 0.8748 - val_accuracy: 0.7360 Epoch 11/100 120/121 [============================>.] - ETA: 0s - loss: 0.2114 - accuracy: 0.9227 Epoch 11: val_accuracy improved from 0.73598 to 0.82944, saving model to best_model.h5 121/121 [==============================] - 6s 48ms/step - loss: 0.2125 - accuracy: 0.9223 - val_loss: 0.5081 - val_accuracy: 0.8294 Epoch 12/100 120/121 [============================>.] - ETA: 0s - loss: 0.2021 - accuracy: 0.9268 Epoch 12: val_accuracy improved from 0.82944 to 0.84813, saving model to best_model.h5 121/121 [==============================] - 6s 52ms/step - loss: 0.2023 - accuracy: 0.9267 - val_loss: 0.4966 - val_accuracy: 0.8481 Epoch 13/100 121/121 [==============================] - ETA: 0s - loss: 0.1775 - accuracy: 0.9363 Epoch 13: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 49ms/step - loss: 0.1775 - accuracy: 0.9363 - val_loss: 1.5754 - val_accuracy: 0.6519 Epoch 14/100 120/121 [============================>.] - ETA: 0s - loss: 0.1478 - accuracy: 0.9443 Epoch 14: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 49ms/step - loss: 0.1481 - accuracy: 0.9441 - val_loss: 0.5397 - val_accuracy: 0.8435 Epoch 15/100 121/121 [==============================] - ETA: 0s - loss: 0.1143 - accuracy: 0.9615 Epoch 15: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 50ms/step - loss: 0.1143 - accuracy: 0.9615 - val_loss: 1.3775 - val_accuracy: 0.6916 Epoch 16/100 120/121 [============================>.] - ETA: 0s - loss: 0.2131 - accuracy: 0.9289 Epoch 16: val_accuracy did not improve from 0.84813 121/121 [==============================] - 6s 47ms/step - loss: 0.2155 - accuracy: 0.9288 - val_loss: 1.4203 - val_accuracy: 0.6799 Epoch 17/100 121/121 [==============================] - ETA: 0s - loss: 0.1556 - accuracy: 0.9480 Epoch 17: val_accuracy improved from 0.84813 to 0.86682, saving model to best_model.h5 121/121 [==============================] - 6s 52ms/step - loss: 0.1556 - accuracy: 0.9480 - val_loss: 0.4611 - val_accuracy: 0.8668 Epoch 18/100 120/121 [============================>.] - ETA: 0s - loss: 0.0900 - accuracy: 0.9701 Epoch 18: val_accuracy did not improve from 0.86682 121/121 [==============================] - 6s 48ms/step - loss: 0.0899 - accuracy: 0.9701 - val_loss: 0.8230 - val_accuracy: 0.7477 Epoch 19/100 120/121 [============================>.] - ETA: 0s - loss: 0.0837 - accuracy: 0.9714 Epoch 19: val_accuracy did not improve from 0.86682 121/121 [==============================] - 6s 49ms/step - loss: 0.0856 - accuracy: 0.9711 - val_loss: 1.8179 - val_accuracy: 0.6379 Epoch 20/100 121/121 [==============================] - ETA: 0s - loss: 0.1230 - accuracy: 0.9561 Epoch 20: val_accuracy did not improve from 0.86682 121/121 [==============================] - 6s 50ms/step - loss: 0.1230 - accuracy: 0.9561 - val_loss: 0.7045 - val_accuracy: 0.8061 Epoch 21/100 120/121 [============================>.] - ETA: 0s - loss: 0.1514 - accuracy: 0.9479 Epoch 21: val_accuracy did not improve from 0.86682 121/121 [==============================] - 6s 48ms/step - loss: 0.1512 - accuracy: 0.9480 - val_loss: 0.6942 - val_accuracy: 0.7944 Epoch 22/100 120/121 [============================>.] - ETA: 0s - loss: 0.0827 - accuracy: 0.9693 Epoch 22: val_accuracy did not improve from 0.86682 121/121 [==============================] - 6s 50ms/step - loss: 0.0827 - accuracy: 0.9693 - val_loss: 0.6314 - val_accuracy: 0.8341 Epoch 23/100 121/121 [==============================] - ETA: 0s - loss: 0.0603 - accuracy: 0.9800 Epoch 23: val_accuracy did not improve from 0.86682 121/121 [==============================] - 6s 49ms/step - loss: 0.0603 - accuracy: 0.9800 - val_loss: 2.7126 - val_accuracy: 0.5047 Epoch 24/100 120/121 [============================>.] - ETA: 0s - loss: 0.0530 - accuracy: 0.9846 Epoch 24: val_accuracy improved from 0.86682 to 0.89720, saving model to best_model.h5 121/121 [==============================] - 6s 49ms/step - loss: 0.0538 - accuracy: 0.9844 - val_loss: 0.3790 - val_accuracy: 0.8972 Epoch 25/100 120/121 [============================>.] - ETA: 0s - loss: 0.1026 - accuracy: 0.9643 Epoch 25: val_accuracy did not improve from 0.89720 121/121 [==============================] - 6s 49ms/step - loss: 0.1034 - accuracy: 0.9641 - val_loss: 1.1961 - val_accuracy: 0.6776 Epoch 26/100 121/121 [==============================] - ETA: 0s - loss: 0.1000 - accuracy: 0.9670 Epoch 26: val_accuracy did not improve from 0.89720 121/121 [==============================] - 6s 48ms/step - loss: 0.1000 - accuracy: 0.9670 - val_loss: 0.5659 - val_accuracy: 0.8621 Epoch 27/100 120/121 [============================>.] - ETA: 0s - loss: 0.0665 - accuracy: 0.9784 Epoch 27: val_accuracy did not improve from 0.89720 121/121 [==============================] - 6s 49ms/step - loss: 0.0669 - accuracy: 0.9782 - val_loss: 0.5263 - val_accuracy: 0.8668 Epoch 28/100 121/121 [==============================] - ETA: 0s - loss: 0.0534 - accuracy: 0.9839 Epoch 28: val_accuracy did not improve from 0.89720 121/121 [==============================] - 6s 48ms/step - loss: 0.0534 - accuracy: 0.9839 - val_loss: 0.5661 - val_accuracy: 0.8668 Epoch 29/100 120/121 [============================>.] - ETA: 0s - loss: 0.0806 - accuracy: 0.9729 Epoch 29: val_accuracy did not improve from 0.89720 121/121 [==============================] - 6s 47ms/step - loss: 0.0826 - accuracy: 0.9724 - val_loss: 1.1458 - val_accuracy: 0.7033 Epoch 30/100 120/121 [============================>.] - ETA: 0s - loss: 0.0935 - accuracy: 0.9682 Epoch 30: val_accuracy did not improve from 0.89720 121/121 [==============================] - 6s 49ms/step - loss: 0.0937 - accuracy: 0.9680 - val_loss: 0.6199 - val_accuracy: 0.8528 Epoch 31/100 120/121 [============================>.] - ETA: 0s - loss: 0.0455 - accuracy: 0.9870 Epoch 31: val_accuracy did not improve from 0.89720 121/121 [==============================] - 6s 47ms/step - loss: 0.0459 - accuracy: 0.9867 - val_loss: 0.6134 - val_accuracy: 0.8458 Epoch 32/100 121/121 [==============================] - ETA: 0s - loss: 0.0428 - accuracy: 0.9857 Epoch 32: val_accuracy did not improve from 0.89720 121/121 [==============================] - 6s 48ms/step - loss: 0.0428 - accuracy: 0.9857 - val_loss: 0.7207 - val_accuracy: 0.8575 Epoch 33/100 121/121 [==============================] - ETA: 0s - loss: 0.0531 - accuracy: 0.9802 Epoch 33: val_accuracy did not improve from 0.89720 121/121 [==============================] - 6s 49ms/step - loss: 0.0531 - accuracy: 0.9802 - val_loss: 0.7820 - val_accuracy: 0.7757 Epoch 34/100 120/121 [============================>.] - ETA: 0s - loss: 0.0499 - accuracy: 0.9828 Epoch 34: val_accuracy did not improve from 0.89720 121/121 [==============================] - 6s 48ms/step - loss: 0.0509 - accuracy: 0.9826 - val_loss: 1.0555 - val_accuracy: 0.7687 Epoch 34: early stopping
Training stopped at epoch 20. The highest validation accuracy was 0.8916 with a validation loss of 0.3837.
Model8 Accuracy vs Epoch curve
plt.plot(history8.history['accuracy'])
plt.plot(history8.history['val_accuracy'])
plt.title('model8 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
The model overfits.
Model8 Evaluation of model model.evaluate using the Test data
# Evaluate the model on the test data
score8 = model8.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score8[0])
print('Test accuracy:', score8[1])
Test loss: 0.9670275449752808 Test accuracy: 0.7873684167861938
# Test Prediction
y_pred = model8.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 13ms/step
# Test Accuracy
accuracyScore8 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore8)
0.7873684210526316
Classification Report
# Compute the classification report
cr8 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr8)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.90 0.92 0.91 50
Fat Hen 0.94 0.96 0.95 48
Shepherds Purse 0.89 0.35 0.50 23
Common wheat 0.80 0.73 0.76 22
Common Chickweed 0.63 0.98 0.77 61
Charlock 0.95 0.90 0.92 39
Cleavers 0.81 0.90 0.85 29
Scentless Mayweed 0.97 0.63 0.77 52
Sugar beet 0.80 0.84 0.82 38
Maize 1.00 0.59 0.74 22
Black-grass 0.39 0.62 0.48 26
Loose Silky-bent 0.80 0.66 0.72 65
accuracy 0.79 475
macro avg 0.82 0.76 0.77 475
weighted avg 0.83 0.79 0.79 475
Overall the accuracy is low at 0.43. We have developed better models earlier.
Scoresdf = Scoresdf.append({'model':'Model8','accuracy':accuracyScore8,'layers':'20','total parameters':'894,028', 'neurons':'256', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Compute Confusion matrix
cm8 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm8, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
From the confusion matrix we can observe that Model8 is unable to predict many classes correctly at all. Overall, this is not a good model.
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
This will be our final model trying to improve our previous models. Again, we will modify the number of layers, filter size and neurons.
Model Implementation
# Intializing a sequential model
model9 = Sequential()
# Adding first conv layer with 64 filters and kernel size 3x3 , padding 'same' provides the output size same as the input size
# Input_shape denotes input image dimension of MNIST images
model9.add(Conv2D(64, (3, 3), activation='relu', padding="same", input_shape=input_shape))
model9.add(Conv2D(64, (3, 3), activation='relu', padding="same"))
# Adding max pooling to reduce the size of output of first conv layer
model9.add(MaxPooling2D((2, 2), padding = 'same'))
model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
model9.add(MaxPooling2D((2, 2), padding = 'same'))
model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
model9.add(MaxPooling2D((2, 2), padding = 'same'))
# flattening the output of the conv layer after max pooling to make it ready for creating dense connections
model9.add(Flatten())
# Adding a fully connected dense layer with 100 neurons
model9.add(Dense(128, activation='relu'))
model9.add(Dropout(0.25))
model9.add(Dense(64, activation='relu'))
# Adding the output layer with 10 neurons and activation functions as softmax since this is a multi-class classification problem
model9.add(Dense(num_classes, activation='softmax'))
# Using Adam Optimizer
opt = Adam()
# Compile model
model9.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model9.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 128, 128, 64) 1792
conv2d_1 (Conv2D) (None, 128, 128, 64) 36928
max_pooling2d (MaxPooling2D (None, 64, 64, 64) 0
)
conv2d_2 (Conv2D) (None, 64, 64, 32) 18464
conv2d_3 (Conv2D) (None, 64, 64, 32) 9248
max_pooling2d_1 (MaxPooling (None, 32, 32, 32) 0
2D)
conv2d_4 (Conv2D) (None, 32, 32, 32) 9248
conv2d_5 (Conv2D) (None, 32, 32, 32) 9248
max_pooling2d_2 (MaxPooling (None, 16, 16, 32) 0
2D)
flatten (Flatten) (None, 8192) 0
dense (Dense) (None, 128) 1048704
dropout (Dropout) (None, 128) 0
dense_1 (Dense) (None, 64) 8256
dense_2 (Dense) (None, 12) 780
=================================================================
Total params: 1,142,668
Trainable params: 1,142,668
Non-trainable params: 0
_________________________________________________________________
Model9 Architecture Visualization
#visualize model9 architecture
plot_model(model9, to_file='model9_plot.png', show_shapes=True, show_layer_names=True)
#visualize model9 architecture using visualkeras
visualkeras.layered_view(model9, legend=True)
Early Stopping
Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history9=model9.fit(X_train,
y_train_onehot,
epochs=100,
batch_size=32,validation_split=0.10,callbacks=[es, mc])
Epoch 1/100 120/121 [============================>.] - ETA: 0s - loss: 2.4315 - accuracy: 0.1240 Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5 121/121 [==============================] - 10s 55ms/step - loss: 2.4312 - accuracy: 0.1237 - val_loss: 2.4521 - val_accuracy: 0.1215 Epoch 2/100 120/121 [============================>.] - ETA: 0s - loss: 2.0708 - accuracy: 0.2891 Epoch 2: val_accuracy improved from 0.12150 to 0.35748, saving model to best_model.h5 121/121 [==============================] - 6s 50ms/step - loss: 2.0710 - accuracy: 0.2893 - val_loss: 1.7058 - val_accuracy: 0.3575 Epoch 3/100 120/121 [============================>.] - ETA: 0s - loss: 1.5718 - accuracy: 0.4453 Epoch 3: val_accuracy improved from 0.35748 to 0.48131, saving model to best_model.h5 121/121 [==============================] - 6s 52ms/step - loss: 1.5718 - accuracy: 0.4453 - val_loss: 1.5090 - val_accuracy: 0.4813 Epoch 4/100 120/121 [============================>.] - ETA: 0s - loss: 1.3208 - accuracy: 0.5378 Epoch 4: val_accuracy improved from 0.48131 to 0.59112, saving model to best_model.h5 121/121 [==============================] - 6s 50ms/step - loss: 1.3205 - accuracy: 0.5378 - val_loss: 1.1847 - val_accuracy: 0.5911 Epoch 5/100 121/121 [==============================] - ETA: 0s - loss: 1.0526 - accuracy: 0.6356 Epoch 5: val_accuracy improved from 0.59112 to 0.62617, saving model to best_model.h5 121/121 [==============================] - 6s 50ms/step - loss: 1.0526 - accuracy: 0.6356 - val_loss: 1.0737 - val_accuracy: 0.6262 Epoch 6/100 121/121 [==============================] - ETA: 0s - loss: 0.8805 - accuracy: 0.6868 Epoch 6: val_accuracy improved from 0.62617 to 0.67290, saving model to best_model.h5 121/121 [==============================] - 6s 51ms/step - loss: 0.8805 - accuracy: 0.6868 - val_loss: 0.9630 - val_accuracy: 0.6729 Epoch 7/100 120/121 [============================>.] - ETA: 0s - loss: 0.7282 - accuracy: 0.7346 Epoch 7: val_accuracy improved from 0.67290 to 0.68224, saving model to best_model.h5 121/121 [==============================] - 6s 50ms/step - loss: 0.7284 - accuracy: 0.7349 - val_loss: 0.9714 - val_accuracy: 0.6822 Epoch 8/100 120/121 [============================>.] - ETA: 0s - loss: 0.5953 - accuracy: 0.7810 Epoch 8: val_accuracy improved from 0.68224 to 0.71262, saving model to best_model.h5 121/121 [==============================] - 6s 53ms/step - loss: 0.5949 - accuracy: 0.7809 - val_loss: 0.8997 - val_accuracy: 0.7126 Epoch 9/100 121/121 [==============================] - ETA: 0s - loss: 0.4768 - accuracy: 0.8095 Epoch 9: val_accuracy improved from 0.71262 to 0.71495, saving model to best_model.h5 121/121 [==============================] - 6s 50ms/step - loss: 0.4768 - accuracy: 0.8095 - val_loss: 1.0595 - val_accuracy: 0.7150 Epoch 10/100 121/121 [==============================] - ETA: 0s - loss: 0.4320 - accuracy: 0.8303 Epoch 10: val_accuracy improved from 0.71495 to 0.73364, saving model to best_model.h5 121/121 [==============================] - 6s 50ms/step - loss: 0.4320 - accuracy: 0.8303 - val_loss: 1.0687 - val_accuracy: 0.7336 Epoch 11/100 120/121 [============================>.] - ETA: 0s - loss: 0.4239 - accuracy: 0.8435 Epoch 11: val_accuracy did not improve from 0.73364 121/121 [==============================] - 6s 50ms/step - loss: 0.4241 - accuracy: 0.8433 - val_loss: 1.0077 - val_accuracy: 0.7313 Epoch 12/100 120/121 [============================>.] - ETA: 0s - loss: 0.3202 - accuracy: 0.8776 Epoch 12: val_accuracy did not improve from 0.73364 121/121 [==============================] - 6s 48ms/step - loss: 0.3203 - accuracy: 0.8773 - val_loss: 1.1118 - val_accuracy: 0.7220 Epoch 13/100 120/121 [============================>.] - ETA: 0s - loss: 0.3107 - accuracy: 0.8802 Epoch 13: val_accuracy improved from 0.73364 to 0.75467, saving model to best_model.h5 121/121 [==============================] - 6s 53ms/step - loss: 0.3113 - accuracy: 0.8799 - val_loss: 0.9295 - val_accuracy: 0.7547 Epoch 14/100 120/121 [============================>.] - ETA: 0s - loss: 0.2699 - accuracy: 0.9031 Epoch 14: val_accuracy did not improve from 0.75467 121/121 [==============================] - 6s 49ms/step - loss: 0.2700 - accuracy: 0.9030 - val_loss: 1.0213 - val_accuracy: 0.7383 Epoch 15/100 120/121 [============================>.] - ETA: 0s - loss: 0.2276 - accuracy: 0.9146 Epoch 15: val_accuracy did not improve from 0.75467 121/121 [==============================] - 6s 51ms/step - loss: 0.2274 - accuracy: 0.9147 - val_loss: 1.1179 - val_accuracy: 0.7336 Epoch 16/100 121/121 [==============================] - ETA: 0s - loss: 0.2142 - accuracy: 0.9254 Epoch 16: val_accuracy did not improve from 0.75467 121/121 [==============================] - 6s 50ms/step - loss: 0.2142 - accuracy: 0.9254 - val_loss: 1.1452 - val_accuracy: 0.7173 Epoch 17/100 120/121 [============================>.] - ETA: 0s - loss: 0.1628 - accuracy: 0.9440 Epoch 17: val_accuracy did not improve from 0.75467 121/121 [==============================] - 6s 50ms/step - loss: 0.1625 - accuracy: 0.9441 - val_loss: 1.0877 - val_accuracy: 0.7360 Epoch 18/100 120/121 [============================>.] - ETA: 0s - loss: 0.1426 - accuracy: 0.9451 Epoch 18: val_accuracy did not improve from 0.75467 121/121 [==============================] - 6s 52ms/step - loss: 0.1425 - accuracy: 0.9452 - val_loss: 1.2663 - val_accuracy: 0.7430 Epoch 18: early stopping
Training stopped at epoch 20. The highest validation accuracy was 0.79907 with a validation loss of 1.145.
Model9 Accuracy vs Epoch curve
plt.plot(history9.history['accuracy'])
plt.plot(history9.history['val_accuracy'])
plt.title('model9 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
The model overfits.
Model9 Evaluation of model model.evaluate using the Test data
# Evaluate the model on the test data
score9 = model9.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score8[0])
print('Test accuracy:', score8[1])
Test loss: 0.9670275449752808 Test accuracy: 0.7873684167861938
# Test Prediction
y_pred = model9.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)
15/15 [==============================] - 0s 14ms/step
# Test Accuracy
accuracyScore9 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore8)
0.7873684210526316
Classification Report
# Compute the classification report
cr9 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)
print('Classification report:\n',cr9)
Classification report:
precision recall f1-score support
Small-flowered Cranesbill 0.90 0.86 0.88 50
Fat Hen 0.73 0.85 0.79 48
Shepherds Purse 0.53 0.35 0.42 23
Common wheat 0.64 0.64 0.64 22
Common Chickweed 0.75 0.92 0.82 61
Charlock 0.77 0.87 0.82 39
Cleavers 0.81 0.86 0.83 29
Scentless Mayweed 0.71 0.65 0.68 52
Sugar beet 0.77 0.63 0.70 38
Maize 0.65 0.50 0.56 22
Black-grass 0.36 0.35 0.35 26
Loose Silky-bent 0.70 0.68 0.69 65
accuracy 0.72 475
macro avg 0.69 0.68 0.68 475
weighted avg 0.72 0.72 0.72 475
Model9 with four convolutional layers indicates that the overall performance is good, with an accuracy of 78%.
Scoresdf = Scoresdf.append({'model':'Model9','accuracy':accuracyScore9,'layers':'14','total parameters':'1,142,668', 'neurons':'64', 'optimizer':'Adam'},ignore_index=True)
Confusion Matrix
# Compute Confusion matrix
cm9 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm9, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()
Model9 performs well but not as good as Model4.
We tried to improve our model by implementing 9 models. One approach to implement a model is by changing the hyperparameters like optimizer, number of neurons, number of filters, adding convolution layers, batchnormalizers, changing the dropout rate, etc.
We performed these to improve our model's performance:
Architecture Modifications: Modifying the architecture of the CNN by adding or removing layers, changing the number of filters or neurons in each layer, or using different activation functions can improve performance. We implemented diverse models this way.
Hyperparameter Tuning: Optimizing the hyperparameters of the model, such as learning rate, batch size, and optimizer, can improve performance. This can be done using techniques such as grid search or random search. We changed the optimizer (Adam, SGD).
Regularization: Regularization techniques such as dropout and L2 regularization can prevent overfitting and improve generalization performance. We performed dropout.
Batch Normalization: Batch normalization is a technique that normalizes the inputs of each layer to prevent internal covariate shift. This can improve performance and reduce the number of epochs needed for training. We used batch Normalization.
Early Stopping: Early stopping is a technique that stops training the model when the validation loss stops improving. This prevents overfitting and improves generalization performance. We used Early stopping.
We didn't implement:
Data Augmentation: Increasing the amount of data available for training the model can improve performance. Data augmentation techniques such as flipping, rotating, scaling, and cropping images can create new variations of the same images and help the model to generalize better. However, this may require computational power so it may be expensive to perform.
Transfer Learning: Pretrained models can be used as a starting point for training a new model. Transfer learning involves using the weights of a model trained on a large dataset, such as ImageNet, to initialize the weights of a new model. This can speed up training and improve performance.
Ensembling: Ensemble methods involve combining the predictions of multiple models to improve performance. This can be done by training multiple CNN models with different hyperparameters or architectures and combining their predictions.
From Model Improvement we came up with 9 different models which are compared below:
Scoresdf
| model | accuracy | layers | total parameters | neurons | optimizer | |
|---|---|---|---|---|---|---|
| 0 | Model1 | 0.730526 | 8 | 7,393,868 | 128 | Adam |
| 1 | Model2 | 0.787368 | 10 | 3,306,188 | 128 | Adam |
| 2 | Model3 | 0.772632 | 10 | 3,306,188 | 128 | SGD |
| 3 | Model4 | 0.705263 | 17 | 834,252 | 128 | Adam |
| 4 | Model5 | 0.755789 | 20 | 821,324 | 256 | Adam |
| 5 | Model6 | 0.421053 | 17 | 26,200,716 | 256 | Adam |
| 6 | Model7 | 0.795789 | 20 | 9,733,804 | 512 | Adam |
| 7 | Model8 | 0.787368 | 20 | 894,028 | 256 | Adam |
| 8 | Model9 | 0.722105 | 14 | 1,142,668 | 64 | Adam |
The model that performed best was Model4 with an overall accuracy of 0.83.
Model4 has 17 layers and trained 834,52 parameters. The fully connected layer had 128 neurons and the optimizer run was Adam.
The precision scores range from 0.48 to 0.95, indicating that the model's ability to correctly identify positive instances varies by class. For example, the model has high precision in identifying Charlock and Cleavers, while it has relatively lower precision in identifying Black-grass and Shepherds Purse.
The recall scores range from 0.50 to 0.97, indicating that the model's ability to correctly identify positive instances also varies by class. For example, the model has high recall in identifying Common Chickweed and Charlock, while it has relatively lower recall in identifying Black-grass and Shepherds Purse.
The F1-scores are a harmonic mean of precision and recall and range from 0.49 to 0.92. The F1-score provides a balance between precision and recall, indicating the overall performance of the model for each class.
The best performing model obtained has 4 Conv2D layers, 128 filters per layer, and 2 Dense layers with 256 neurons each. This model achieved an accuracy of 83% on the test set, which is a significant improvement from the baseline model.
We could further try to improve the model's performance by trying mostly:
Increase the size of the training dataset by collecting more images or using data augmentation techniques to generate new images from the existing ones.
Try using pre-trained models such as VGG, ResNet or Inception and fine-tune them on your dataset.
Continue to experiment with different hyperparameters such as batch size, number of epochs, and regularization techniques to find the best combination for your model.
We should notice that building a good performing CNN model requires a lot of experimentation and fine-tuning.
Additionally keep in mind:
Overall, our model performs well but it needs to be fine tuned for the classes that are most difficult to classify, in our case, Black grass.
